I think it should be like this:

- on mousedown save the current mouse position
- on mousemove compare the current mouse position with the saved one.
  This is the offset, and this doesn't have anything to do with the absolute (or already moved) position of the object itself.