When viewentity is set, use the view_ofs of that entity instead of the client.  I should have done this initially because it seems to make the most sense.