A method of forming a three-dimensional memory device, comprising: forming an alternating stack of insulating layers and sacrificial material layers over a substrate; forming a memory opening through the alternating stack: forming annular recesses around the memory opening by laterally recessing the sacrificial material layers with respect to the insulating layers; forming annular metal portions over recessed sidewalls of the sacrificial material layers within each of the annular recesses through the memory opening; selectively forming discrete annular backside blocking dielectrics Z on inner sidewalls of the annular metal portions; forming a memory stack structure on inner sidewalls of the annular backside blocking dielectrics in the memory opening, wherein the memory stack structure comprises a memory film and a vertical semiconductor channel contacting an inner sidewall of the memory film; and replacing the sacrificial material layers with electrically conductive layers (46, 346), whereby each of the annular metal portions contacts a cylindrical sidewall of a respective one of the electrically conductive layers, characterised in that the discrete annular backside blocking dielectrics on inner sidewalls of the annular metal portions are made by a selective dielectric material deposition process that grows a dielectric material of the annular backside blocking dielectric without growing the dielectric material from surfaces of the insulating layers. forming an alternating stack of insulating layers and sacrificial material layers over a substrate; forming a memory opening through the alternating stack: forming annular recesses around the memory opening by laterally recessing the sacrificial material layers with respect to the insulating layers; forming annular metal portions over recessed sidewalls of the sacrificial material layers within each of the annular recesses through the memory opening; selectively forming discrete annular backside blocking dielectrics Z on inner sidewalls of the annular metal portions; forming a memory stack structure on inner sidewalls of the annular backside blocking dielectrics in the memory opening, wherein the memory stack structure comprises a memory film and a vertical semiconductor channel contacting an inner sidewall of the memory film; and replacing the sacrificial material layers with electrically conductive layers (46, 346), whereby each of the annular metal portions contacts a cylindrical sidewall of a respective one of the electrically conductive layers, characterised in that the discrete annular backside blocking dielectrics on inner sidewalls of the annular metal portions are made by a selective dielectric material deposition process that grows a dielectric material of the annular backside blocking dielectric without growing the dielectric material from surfaces of the insulating layers. forming annular recesses around the memory opening by laterally recessing the sacrificial material layers with respect to the insulating layers; forming annular metal portions over recessed sidewalls of the sacrificial material layers within each of the annular recesses through the memory opening; selectively forming discrete annular backside blocking dielectrics Z on inner sidewalls of the annular metal portions; forming a memory stack structure on inner sidewalls of the annular backside blocking dielectrics in the memory opening, wherein the memory stack structure comprises a memory film and a vertical semiconductor channel contacting an inner sidewall of the memory film; and replacing the sacrificial material layers with electrically conductive layers (46, 346), whereby each of the annular metal portions contacts a cylindrical sidewall of a respective one of the electrically conductive layers, characterised in that the discrete annular backside blocking dielectrics on inner sidewalls of the annular metal portions are made by a selective dielectric material deposition process that grows a dielectric material of the annular backside blocking dielectric without growing the dielectric material from surfaces of the insulating layers.