The movement of a selected hole is relative to the segment that you select it at. Using an example of a simple cylindrical hole through a cube, select the hole by picking a segment on the face that the hole penetrates. Don't use a segment along the hole's length. I'm sure the behaviour is different for uniquely shaped holes in a more complex object, but you will see that the behaviour can be predicted. Because of this, when you move holes, the system generates a temporary arbitrary reference plane, whose axis orientation and coordinates it uses to move your hole.
More explanation is in order.
We shall do it by describing how things work. Here is why. Because this temporary reference plane is oriented parallel to your rectangular wall, you are able to move the hole where you want. When you snap, the hole will move in the direction of the axes, which appears perfectly normal. When you move this hole, if you click on a segment that is not parallel to the rectangular shape of the wall, the temporary reference plane is generated in the direction of that segment.