A method of forming a three-dimensional memory device, comprising: forming a stack of alternating layers comprising first material layers and second material layers over a substrate; forming a memory opening through the stack; forming a memory film (50, 50') and a semiconductor channel in the memory opening; forming backside recesses by removing the second material layers selective to the first material layers and the memory film, wherein an outer sidewall of the memory film is physically exposed within each backside recess, wherein the physically exposed outer surfaces of the memory film are -NH terminated surfaces, and wherein the first material layers have -OH terminated insulator surfaces in the back side recesses; cleaning the physically exposed outer surfaces of the memory film and surfaces of the first material layers with hydrofluoric acid; after said cleaning, exposing the -NH terminated surfaces and the -OH terminated insulator surfaces to a silylation agent containing an -Si(CH 3) 3 group to convert the -OH terminated insulator surfaces to -OSi(CH 3) 3 terminated insulator surfaces while the -NH terminated surfaces maintain -NH termination; drying the -OSi(CH 3) 3 terminated insulator surfaces and -NH terminated surfaces of the memory film with isopropyl alcohol after exposure to the silylation agent; selectively growing a silicon-containing semiconductor portion on the -NH terminated surface of the memory film within each backside recess, while silicon-containing semiconductor portions do not grow from the -OSi(CH 3) 3 terminated insulator surfaces; converting an outer portion of each silicon-containing semiconductor portion into a dielectric material portion, wherein a remaining portion of each silicon-containing semiconductor portion constitutes a floating gate electrode for the three-dimensional memory device; and subsequently forming conductive material layers by depositing a conductive material within the backside recesses; wherein the -NH terminated surfaces are surfaces of a silicon nitride layer; the -OSi(CH 3) 3 terminated insulator surfaces are surfaces of silicon oxide layers; and the silicon-containing semiconductor portion is an amorphous silicon portion. forming a stack of alternating layers comprising first material layers and second material layers over a substrate; forming a memory opening through the stack; forming a memory film (50, 50') and a semiconductor channel in the memory opening; forming backside recesses by removing the second material layers selective to the first material layers and the memory film, wherein an outer sidewall of the memory film is physically exposed within each backside recess, wherein the physically exposed outer surfaces of the memory film are -NH terminated surfaces, and wherein the first material layers have -OH terminated insulator surfaces in the back side recesses; cleaning the physically exposed outer surfaces of the memory film and surfaces of the first material layers with hydrofluoric acid; after said cleaning, exposing the -NH terminated surfaces and the -OH terminated insulator surfaces to a silylation agent containing an -Si(CH 3) 3 group to convert the -OH terminated insulator surfaces to -OSi(CH 3) 3 terminated insulator surfaces while the -NH terminated surfaces maintain -NH termination; drying the -OSi(CH 3) 3 terminated insulator surfaces and -NH terminated surfaces of the memory film with isopropyl alcohol after exposure to the silylation agent; selectively growing a silicon-containing semiconductor portion on the -NH terminated surface of the memory film within each backside recess, while silicon-containing semiconductor portions do not grow from the -OSi(CH 3) 3 terminated insulator surfaces; converting an outer portion of each silicon-containing semiconductor portion into a dielectric material portion, wherein a remaining portion of each silicon-containing semiconductor portion constitutes a floating gate electrode for the three-dimensional memory device; and subsequently forming conductive material layers by depositing a conductive material within the backside recesses; wherein the -NH terminated surfaces are surfaces of a silicon nitride layer; the -OSi(CH 3) 3 terminated insulator surfaces are surfaces of silicon oxide layers; and the silicon-containing semiconductor portion is an amorphous silicon portion.