Your GPU is orthogonal to your motherboard, so be cautious not to lose its angle if you detach it. If you lean it in a different direction, you risk injuring it.