Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new, practically motivated semi-supervised setting, where the agent can access both labeled trajectories and unlabelled trajectories that do not include the actions of the trajectories. A model is trained to give actions for unlabeled data and then the offline RL method is trained by the whole dataset. Experiments based on D4RL dataset show the good performance of the proposed method.
strength_and_weaknesses: The paper is well-written and easy to follow. The setting is novel and meaningful since there are a lot of unlabeled data like videos in real-world scenarios.  The experiments are designed carefully to illustrate the influence of data with different qualities. For the weakness, Iwould like to ask some questions:
1. For IDM, the length of the input, k, could be changed, and k=1 in the paper. So the input of the model is four states (s
_t-1,..., s_t+1). How does the model encode these states?
2.  The experiments only use the expert dataset, which means most trajectories are good. D4RL also provides random and medium-level dataset. As your claim in the paper, the quality of the data has a huge influence on the performance. Is there any analysis based on these data? Could we say that the method also needs high-quality data for both labeled and unlabeled data to achieve good performance?
clarity,_quality,_novelty_and_reproducibility: The detailed dataset and code of the method are not provided.
summary_of_the_review: See the above-mentioned questions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a method for semi-supervised learning in the offline RL setting where the unlabelled part of the dataset consists of action-free state trajectories and the labelled part consists of the full trajectories. They use an inverse dynamics model to learn actions that give rise to state transitions and use the learned model to inject labels for unlabelled data and perform offline learning using classic model-free offline RL algorithms such as CQL.
strength_and_weaknesses: Weaknesses:
- My main concern is with the technical contribution of this paper. Considering the inverse dynamics model as the crucial technical novelty, I am not sure the method given in Eq. 1 is the best, and at the very least, some further evaluations and arguments for its design choice is needed. For one, the covariate matrix with k>0 would seem to break the markov property of the RL setting. Furthermore, it is not justified why this choice was made beyond the empirical validation, which is not enough either in my opinion. I am not convinced that this extension of IDM significantly contributes to the overall goal of taking advantage of semi-supervision, as compared to the fully unsupervised case. In particular, it would have been interesting to see whether semi-supervision in this regime would help when the data split is within trajectories and not between trajectories. 
- Along the lines of the above, I think an analysis of why only a single labelling round was used instead of the conventional self-training paradigm of retraining per round could have been included to make the paper stronger. In particular, I assume it could be possible to provide a more detailed analysis of how well the inverse dynamics model learns when compared to the ground truth data on the log-likelihood of the multivariate Gaussians used to estimate them since you have access to those data labels. 
- As it stands, this paper's main contributions are a careful study of the different considerations one should take when doing semi-supervised offline RL. It has a good experimental validation of how performant value-based and BC methods. However, it does not make enough technical contribution to take advantage of the specific literature in the semi-supervised learning setting or does not justify its design choices well. 
clarity,_quality,_novelty_and_reproducibility: This paper is clearly written, with some minor typos throughout. Overall, my main concern is with the novelty and quality of the proposed algorithm. The results should be reproducible.
summary_of_the_review: Overall, I lean towards rejecting this paper. My reasoning is that while it is tackling a notable practical problem in the RL setting, there is not enough technical contribution nor is it polished enough in its exposition of the design choices for me to recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper provides an empirical study about a setting where a portion of the offline RL dataset doesn't include actions. To exploit action-missing data, this work proposes to learn an inverse dynamics model on data with actions to generate proxy actions from state transitions. A set of empirical studies on d4rl gym-locomotion control are provided to give insights into how such semi-supervised learning will be helpful for the performance of the final policy. The ablation studies show the proposed semi-supervised method is particularly helpful when the action-missing data is of high quality and labelled data is of lower quality.
strength_and_weaknesses: Strength:
1. The setting is well-motivated and has decent potential for real-world applications. 
2. The empirical studies are thorough and in particular, ablation over the quality of data is well-done.

Weaknesses:
1. The proposed method is quite standard in online settings. The claimed novelty (multiple transitions as input to the inverse model) is more of technical detail and the reason why it is helpful for the Markovian setting is not explained properly.
2. Most of the experiment results in the paper are based on the medium-expert dataset of d4rl gym-locomotion. This raises questions about whether the conclusion can be generalised to the setting with more diverse data. For medium-expert, the behaviour policy is basically a mixture of two policies (medium level and expert level). If the trajectories are split into multiple groups according to the returns, it's likely that the root cause of the varied returns is initial states rather than the quality of the policy. One clue is the experiments on medium-replay data in Figure C.2 show for hopper and walker2d, the quality of the unlabelled data plays a much more important role than the case in the medium and medium expert.

Minor issues
1. Is "label" a proper word to replace action? My impression is unlabelled data in offline RL is more about reward-missing data. The intuition behind that is humans can label rewards to trajectories easily because it's a scale but "labelling" actions seems like very difficult.
2. There are some obvious grammatical errors in the paper: e.g. "we are mainly interested in the case **only** a **significant majority** of the trajectories in the offline trajectories are **unlabeled**." "How can we utilize the unlabelled data for improving **the performance offline RL algorithms**?"

clarity,_quality,_novelty_and_reproducibility: The paper is easy to follow in general and the writing is clear.
The quality of empirical evaluations is fine.
The methodology originality is low but the empirical study on the offline RL setting is novel.
summary_of_the_review: The paper studies an interesting setting for offline RL which has significant practical value. The study of different scenarios of data availability and the quality of the data is a timely topic for offline RL research.
However, my main concern about this paper is it's not very informative. The whole paper looks more like a technical report about applying the inverse dynamics model to the d4rl dataset and I'm struggling to figure out what's the key takeaway. 
The authors summarised three of their key findings in the introduction:
1. One interesting claim of the paper is SS-ORL agent works well in the setting with lower-quality labelled and high-quality unlabelled data. But I'm not sure if such a claim can generalize to the setting when the policies are more diverse than a single or a mixture of two unimodal policies, as I argued in the second point of the weakness. 
2. "When the labelled data quality is high, utilizing unlabelled data does not bring significant benefits." This might just reveal that most of the data from the d4rl gym-locomotion are redundant because of the low diversity of the policy?
3. "CQL and TD3BC are less sensitive to the action labels compared to DT." This is not very obvious to me by just looking at Figure 4.1 and Figure 4.2. It would be better to have quantitative results if that's true.

Given the reasons above, I cannot recommend acceptance at the current stage. But I may change my mind if the authors or other reviewers can convince me about point 1, or remind me why point 2 and 3 are significant enough to accept the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses offline reinforcement learning where two sets of trajectories are available: one with fully labeled state-action-reward, and one only with state-reward.  The idea is to learn an inverse-dynamic model from the labeled trajectories so that the unlabeled trajectories can be imputed.  By applying standard offline RL algorithms to the new trajectory set, superior empirical performance is demonstrated on some D4RL datasets.

Overall, the paper is well written, and addresses an important problem.  However, the empirical results are still not convincing enough, especially in the variety of experiment settings, datasets, and return distributions.  The rebuttals have alleviated some concerns, but the paper still needs a new round a revision and review.

justification_for_why_not_higher_score: The empirical results are still not convincing enough, especially in the variety of experiment settings, datasets, and return distributions.  The rebuttals have alleviated some concerns, but the paper still needs a new round a revision and review.
Decision: Reject