Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper introduces the solution of an important task: hierarchical concept learning(or temporal abstractions) from demonstration data. Specifically, this paper considers 1) unsupervised setting; 2) the hierarchy of concepts, and conducts experiments in two datasets. However, there are some points in the experiment section to be discussed. 

The pros of this paper include:
- The task setting of this paper is important and applicable. Especially jointly learning concepts from frames and instructions while considering the hierarchy.
- The proposed model utilizes multiple regeneration module for the concept learning, which may also help for other hierarchical-aware tasks.

The cons include (In my opinion, the main weakness is the experiments):
- Some design of model (e.g., Traversing across the concept hierarchy, Observation, and Instruction Regeneration, etc) are not fully estimated in this section: are they really useful? which cases did they corrected? The experiments make the previous model design somewhat difficult to evaluate.
- The selected baselines are also weak. Even the **Random**(Random seg) and **EQUALDIV**(Equal Divided) perform much better than **GRU-supervised**. I believe more powerful baselines should be proposed, otherwise, the contribution of the paper is hard to be recognized.
review_title: The paper provides some insights of an important task, but some details can be further improved.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper addresses a relatively new topic to learn the hierarchical concepts in videos and commentary in an unsupervised manner. The authors proposed a hierarchical and transversing encoder-decoder network architecture to tackle the problem, where the pre-trained ResNet32 and BERT are used as feature extractors, transformers and GRUs serve as the concept encoders and decoders. As the network is unsupervised, the starting time and ending time of each concept is quite crucial and the problem was tackled by training using a soft-DTW loss. A metric (time-wrapped IoU) is also proposed for quantitative evaluation. The experiments indicate the effectiveness of the network and the network was tested in different datasets and scenarios.

I rate the paper as weakly accept and here are some questions.
1. The lower-level concepts and upper-level concepts in both visual and language are a little vague without clear definitions. How do you define the level of concepts and how did you balance the levels in video and language? For example in Figure 5, the results without commentary (A) look more reasonable for z^H_1 to contain all the steps for a higher-level concept (say, pan heating) but the model with commentary (B) split the concepts mainly according to the language descriptions. This may also influence the evaluation scores as different work may have different "flavors" to select concepts and the starting/ending time could be different. 
2. Besides, since your network is trained across the video and language, how did you train the network without commentary part?
3. You used two different evaluation metrics in two different datasets, which seems that you tend to focus on different aspects in different datasets. This is a bit of bias. Besides, the comparison with other work in the second dataset (Chess Openings) is not provided. 
review_title: Unsupervised encoder-decoder network in a relatively new topic

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper addresses the problem of extracting a hierarchy of concepts in an unsupervised way from demonstration data. The authors present a Transformer-based concept abstraction architecture called UNHCLE and show how it discovers meaningful hierarchies using datasets from Chess and Cooking domains. In particular, the model is designed to function without specific temporal supervision, which makes it potentially practical for real-world applications. 

Pros.
+ problem clear stated with potential practical implications
+ experiments on real-world video datasets

Cons.
- The results are evaluated using a time-warping based metric. I am not convinced this metric provides a faithful assessment of the capability of the model. The exact temporal boundaries are also important in discovering the specific concept and ground them back to the visual observations. TW-IoU will not take this into consideration. 
- The novelty of the proposed model is limited. The two-level hierarchy has been seen a lot in relevant fields such as video recognition. So the concept by itself is not new. Introducing a two-level hierarchy into concept learning is also not new. In fact, I find it somewhat arbitrary that the authors decide to set the number of hierarchies to two. The cooking video dataset does not have such annotations so hypothetically the one can set an arbitrary number for the levels of hierarchy. Can the authors explain why this two-level hierarchy is chosen and how it can generalize to other tasks?
- Comparison to works in unsupervised clustering, video segmentation, and change point detection. While the authors form this work as concept learning, the problem nevertheless relates to these research topics. The authors may need to look upon those for a better variety of baselines and also evaluation metrics. 
- Experiment results are weak. The proposed model's performance does not exceed the simple equal division baseline a lot when 64 frames are sampled for each video.
review_title: Somewhat limited approach and experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a method to unsupervisedly discover structure in unlabeled videos, by finding events at different temporal resolutions.

### Strengths:
- The paper focuses on the important problem of exploiting weakly labeled video data, by exploiting its structure, for example by recovering temporal structure in an autoencoder fashion.
- Use of multiple modalities to cross-supervise each other.
- Code is available.

### Weaknesses:
- The concept of hierarchy is not well defined or well motivated. While most hierarchical papers refer to hierarhies of concepts, the hierarchy considered in this paper is much weaker as a hierarchy, and it refers to subactions within longer actions (not actions that are specific instances of more abstract actions). While this way of understanding hierarchy can be valid, it is never explained or motivated in the paper, or even compared to the standard way of understanding it. 
- The overall motivation for the method has a lot of gaps. For example:
	- Regeneration of low level concepts from high level concepts: what are we expecting from a network that moves from a "high in the hierarchy" concept to a "low in the hierarchy" concept? Should we expect the network to randomly select one subconcept? The specific information is not there, the problem is ill-defined (for example, we can go from "cat" to "animal", but not from "animal" to "cat"). How are we expecting any reconstruction? The paper does not provide any justification or intuition.
	- Why are the authors using those specific pairs in the L_dyn term? (last line of page 4 -- I suggest adding equation numbers). Apart from no motivation, there are no ablations showing that those are the correct pairs.
	- Why is the "low" case even necessary? Wouldn't it be possible to train only with the "high" one? This would probably imply rethinking some of the losses, but overall the method would look very similar. This is, the idea of hierarchy would disappear, but this idea is not used in the experiments anyway.
	- What is the motivation for the two modalities? I can understand it can help, but it is not central to the method. This is not necessarily bad, but it requires some explanation.
- The explanation revolves around demonstration data. It is unclear why demonstration data is important for this method, and why it is not general for any human action. The introduction explaining demonstrations in a robotics scenario does not feel related to the content of the paper. For example, a lot of stress is given to "agents" interacting in "environments".
- Some terms introduced in the paper would benefit from a change. For example, a "concept" in the paper is actually an "event", not a "concept". This is more in line with the literature, for example the dataset they use labels that as "event".
- Results on chess are hard to believe. Do the authors think that the system has really learned (unsupervisedly) to identify interesting openings? It could instead be learning strong biases like length of openings in the dataset.
- Quantitative results are not convincing. How does FLAT and even the supervised method perform much worse than the two trivial baselines (random and equal division)? Does FLAT use text data?
- Conceptual assertions that I do not believe to be true: "under a concept, the sequence of frames is nearly deterministic". This is not true, there are nearly infinite ways of having a sequence of frames (video clip) depicting how to crack an egg, for example. Different background, different way of performing the action, different elements in the scene, speed of the action, point of view, etc. This is related to the "regeneration" point above.

### Additional comments and questions:
- Figure 2 is hard to understand. What is it exactly representing? What should we learn from it?
- Does the algorithm have any "motivation" to not predict always a single concept per sequence?
- What is the relationship between u and v?
- Have you tried smaller networks? 8 layers and 8 heads just for the Encoder seem like a very big model for such a relatively simple text.
- In the first paragraph of page 8 the paper mentions that there is only marginal improvement in low level concepts. How are these evaluated? As far as I could understand, there was only GT available for the high level ones.

### Final recommendation
Overall, I believe the paper as it stands is not ready to be presented to ICLR and I recommend a rejection.
review_title: Recommend rejection

Meta-review: Although the paper studies a relevant and important topic, which is about learning of hierarchy of concepts in an unsupervised manner, the reviewers raised several critical concerns. In particular, although the hierarchical structure of concepts is the key idea in this paper, the concept of hierarchy itself is not well explained. How to define the hierarchical level of concepts should be carefully and mathematically discussed. In addition, empirical evaluation is not thorough as reviewers pointed out. Although we acknowledge that the authors addressed concerns by the author response, newly added results are still confusing and more careful treatment is needed before publication. I will therefore reject the paper. 

This work reminds me the the topic called "formal concept analysis" (e.g. see [1]), which mathematically defines concepts as closed sets and constructs a hierarchy of concepts in an unsupervised manner. This method can be viewed as co-clustering and also has a close relationship to closed itemset mining. This approach is used in machine learning (e.g. [2]). I think it is beneficial for the authors to refer such existing and well-established approaches to elaborate this work further.

[1] Davey, B.A., Priestley, H.A.: Introduction to Lattices and Order, Cambridge Univ. Press (2002)  
[2] Yoneda, et al., Learning Graph Representation via Formal Concept Analysis, 	arXiv:1812.03395 
Decision: Reject