This should probably get put back into a dict called retry_options, just for consistency, right?

There is also something missing here I think.  For some reason, we're getting or expecting a TaskRetryOptions object here, yet we only have one half the encoding.  So it means that sometimes this will be a dict, sometimes a TaskRetryOptions object, etc....  We just need to be sure that it is always something consistent.

I suspect this will actually blow up in some cases where there are options but it has never been cast `to_task`, or some variation thereof.