Sometimes, the energy won't go back in. This usually means you've put too much of the energy you're using in, and not enough of the crystal's own energy. Bring it back to the brain, and remix. Switch hands on the crystal, then visualize that handy hose again, but this time flow backwards, through the door. There will come a point, especially early on in your practice, that the crystal is full when you have energy left over.
Doors have their bursting point, and that's a wonderful way to destroy a good crystal.
OK, it's later. I keep saying that, but it bears repeating. The one I use is simply grounding it. I hope you find this helpful to you.