A three-dimensional memory device, comprising: a P-type doped region (102, 202) of a substrate; an N-type doped semiconductor layer (104, 230) on the P-type doped region (102, 202); a memory stack (106, 234) comprising interleaved conductive layers and dielectric layers on the N-type doped semiconductor layer (104, 230); a channel structure (112, 214) extending vertically through the memory stack (106, 234) and the N-type doped semiconductor layer (104, 230) into the P-type doped region (102, 202) wherein the channel structure (112, 214) comprises a memory film (114, 216) and a semiconductor channel, and part of the semiconductor channel along the sidewall of the channel structure (112, 214) is in contact with the N-type doped semiconductor layer (104, 230); an N-type doped semiconductor plug (128, 232) extending vertically into the P-type doped region (102, 202); and a source contact structure extending vertically through the memory stack (106, 234) to be in contact with the N-type doped semiconductor plug (128, 232), wherein a lateral dimension of the N-type doped semiconductor plug (128, 232) is greater than a lateral dimension of the source contact structure, wherein a lateral dimension of a portion of the N-type doped semiconductor plug (128, 232) surrounded by the P-type doped region (102, 202) is greater than a lateral dimension of a portion of the N-type doped semiconductor plug (128, 232) surrounded by the N-type doped semiconductor layer (104, 230), wherein a lateral dimension of N-type doped semiconductor plug (128, 232) is the minimum lateral dimension along the vertical direction, and a lateral dimension of source contact structure is the maximum lateral dimension along the vertical direction, such that any lateral dimension of N-type doped semiconductor plug (128, 232) is greater than any lateral dimension of the source contact structure. a P-type doped region (102, 202) of a substrate; an N-type doped semiconductor layer (104, 230) on the P-type doped region (102, 202); a memory stack (106, 234) comprising interleaved conductive layers and dielectric layers on the N-type doped semiconductor layer (104, 230); a channel structure (112, 214) extending vertically through the memory stack (106, 234) and the N-type doped semiconductor layer (104, 230) into the P-type doped region (102, 202) wherein the channel structure (112, 214) comprises a memory film (114, 216) and a semiconductor channel, and part of the semiconductor channel along the sidewall of the channel structure (112, 214) is in contact with the N-type doped semiconductor layer (104, 230); an N-type doped semiconductor plug (128, 232) extending vertically into the P-type doped region (102, 202); and a source contact structure extending vertically through the memory stack (106, 234) to be in contact with the N-type doped semiconductor plug (128, 232), wherein a lateral dimension of the N-type doped semiconductor plug (128, 232) is greater than a lateral dimension of the source contact structure, wherein a lateral dimension of a portion of the N-type doped semiconductor plug (128, 232) surrounded by the P-type doped region (102, 202) is greater than a lateral dimension of a portion of the N-type doped semiconductor plug (128, 232) surrounded by the N-type doped semiconductor layer (104, 230), wherein a lateral dimension of N-type doped semiconductor plug (128, 232) is the minimum lateral dimension along the vertical direction, and a lateral dimension of source contact structure is the maximum lateral dimension along the vertical direction, such that any lateral dimension of N-type doped semiconductor plug (128, 232) is greater than any lateral dimension of the source contact structure.