Repository: pytorch/rl

Issue:
[Feature Request] Why have mutually exclusive terminated and truncated?
truncated and terminated seem to be mutually exclusive in the design of TorchRL. As in this line
https://github.com/pytorch/rl/blob/1bd5ec640a4a37cbb95c402df91b199c9a8b1736/torchrl/envs/transforms/transforms.py#L5183
Is that necessary? IMO this should not be enforced and the truncated signal can still safely be set while the terminated signal is true, which will allow a more faithful status of the trajectory (it was supposed to be truncated but it was incidentally terminated).
This should be safe as components that read terminated (e.g., values estimators) don't need truncated and should ignore it (truncated sets done and then becomes useless in that case.)

This would allow for example to set the truncation key to terminated when one wants to consider finite-horizon environments.
`StepCounter(truncated_key="terminated")`.
 An alternative would be to have something like a `FiniteHorizonTransform` that sets termination based on a step limit.

Happy to hear your thoughts.


Discussion:
Sure I'm open to this change. I recall having seen something exclusive like that somewhere in gymnasium but I can't find it now so it may just be my imagination...
If StepCounter does the job OOB do we need a new transform? Transforms are costly (in terms of maintenance but also in pure monetary terms since we need to test them in every PR and tests take a long time bc we need to check they work in parallel envs). My take would be to find a way of documenting this in a very clear manner and use StepCounter instead. 
Alternatively we could write `FiniteHorizonTransform` as a thin wrapper around `StepCounter` with the truncated key predefined as you mentioned.
If truncated and terminated don't need to be exclusive (would need to remove that line from the StepCounter and ideally elsewhere too to remove the design choice), then `StepCounter(truncated_key="terminated")` would work OOB and that would be enough yes. It's also better than creating another transform.
@matteobettini this could be of interest btw

Resolving patch:
diff --git a/torchrl/envs/transforms/transforms.py b/torchrl/envs/transforms/transforms.py
--- a/torchrl/envs/transforms/transforms.py
+++ b/torchrl/envs/transforms/transforms.py
@@ -5163,11 +5163,10 @@ def _reset(
     def _step(
         self, tensordict: TensorDictBase, next_tensordict: TensorDictBase
     ) -> TensorDictBase:
-        for step_count_key, truncated_key, done_key, terminated_key in zip(
+        for step_count_key, truncated_key, done_key in zip(
             self.step_count_keys,
             self.truncated_keys,
             self.done_keys,
-            self.terminated_keys,
         ):
             step_count = tensordict.get(step_count_key)
             next_step_count = step_count + 1
@@ -5178,9 +5177,12 @@ def _step(
                 truncated = truncated | next_tensordict.get(truncated_key, False)
                 if self.update_done:
                     done = next_tensordict.get(done_key, None)
-                    terminated = next_tensordict.get(terminated_key, None)
-                    if terminated is not None:
-                        truncated = truncated & ~terminated
+
+                    # we can have terminated and truncated
+                    # terminated = next_tensordict.get(terminated_key, None)
+                    # if terminated is not None:
+                    #     truncated = truncated & ~terminated
+
                     done = truncated | done  # we assume no done after reset
                     next_tensordict.set(done_key, done)
                 next_tensordict.set(truncated_key, truncated)


Test patch: