Oops! When accounting for mouse offsets as a result of a moved view port, I was using dst_rect when I should have used src_rect.