The endoplasmic reticulum (ER) is the organelle critical for endocytosis and exocytosis to occur because the vesicle forms in the ER.