Got movement working. Involved some trickery around taking the previous mouse position's world coordinates and subtracting that from the current mouse position's world coords.