When you fall into a hole you just keep on picking up speed until you reach your maximum velocity or hit bottom. The reason you keep falling in a normal hole is that gravity is continuing to pull you in the direction of the earth's center. In our imaginary scenario, once you passed the midpoint of the earth, you would then be fighting gravity. All the speed you picked up on the way down would slowly diminish as you rose up the other side. It couldn't really be the case that you'd pick up exactly as much speed as would be needed to reach the same spot on the other side.
Some energy always finds a way to escape.
Think of swinging on a swing. If you just push off once and then sit still, the swing doesn't keep going back and forth forever. Gradually it slows down. It's the same situation when you step into that hole. Down you go, and when you come out the other side, your feet will stop at slightly less than ground-level.