Only encode TaskRetryOptions in to_task

Ensure that we only create and populate a TaskRetryOptions right before
we create the Task object.

Add tests to cover more cases.