Don't override the lookat positions we receive for others.