Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The idea of employing a tensor method to generalize NAS seems to be interesting. However, this paper suffers from a few problems: 
1. The presentation of this paper requires improvement. There are some confusion:

- This paper has not clearly described the difference with previous works. Specifically, although this paper claims to give a unified method, it is difficult to distinguish what have been unified. And in the 10-th line of the Abstract, what is the concrete description of "the above problems"?
- In Figure 2,  whether $\alpha^{(1)}$ is the $\theta^{1}$? If yes, in Figure 2(b), it would be better to use $\theta^1$ to replace $\alpha^{(1)}$ directly. And in the caption of Figure 2, what does "$N_1(t)/N_2(t)$" mean, a division or "$N_1(t)$ or $ N_2(t)$"?
- In Algorithm 1, what is the meaning of $R_{N_1(t)}$? And how to choose the value of $R_{N_1(t)}$?
- In Section **PROPOSED METHOD**, how to transform a supernet to a tensor graph?  It is highly suggested that adding more details about the transforming process.

2. The experimental results are not good enough. For example,  in results on Cifar10, there are a number of competitive methods which obtained better results of the proposed one. 

| Model                                           | Err.      | #Params | Cost(GPU days) |
| ----------------------------------------------- | --------- | ------- | -------------- |
| TRACE(This paper)                               | 2.78±0.12 | 3.3     | 0.6            |
| P-DARTS + cutout(Xin Chen et. al.. ICCV 2019.)  | 2.5       | 3.4     | 0.3            |
| PC-DARTS + cutout(Yuhui Xu et. al.. ICLR 2020.) | 2.57±0.07 | 3.6     | 0.1            |

3. The writing requires significant proofreading. There are plenty of grammar errors：

- In the 2-nd line of Section 2.2, "Tensor methods have found wide applications" -> "... have been  found in..."
- In the 5-th line of Section 3.1, "Note that a subnet P can be distinguished by its choices on each edge, we propose...." -> "... each edge, and we propose...."
- In page 2,  "it also emerge in"
- etc..
review_title: This paper proposed a new way to unify the searching strategy for NAS

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

This paper proposes a new and general formulation for supernet, which encodes supernet with tensor network(TN). Based on TN, the topology of supernet can be encoded. Besides, this paper proposes a corresponding algorithm to solve the search problem. 

Reasons for score:

Overall, I vote for accepting marginally. My major concern is about the clarity of the paper and some additional experiments (see cons below). Hopefully the authors can address my concern in the rebuttal period.

Pros:

1.In this paper, the authors introduce a new and general formulation for supernet. Due to its generalization, supernet-based NAS methods with it can be applied to many deep learning tasks. In addition, this formulation can encode the topology of supernet, which benefits the network search.

2.Based on this formulation, this paper proposes a corresponding search algorithm. This algorithm can solve supernet search problem for both deterministic formulation and stochastic formulation. 

3.Experiments are well thought out and highlight the key advantages of the method over other NAS methods. 

Cons:
1.For recent NAS methods, they all validate their performances on ImageNet. In the experiments, there is only a experiment on ImageNet-16-120. Could you provide an experiment in weight-sharing setting on ImageNet dataset?

2.For Table 3, it is a must to compare with more NAS methods, such as PDARTS and DropNAS. In this way, the effective of the proposed method can be validated fully.

3.In section 3.2, $R^{n}$ in $\mathcal{T}_{i-}$ is used without any definition. Are there some hints for it? If not, it would lead to some misunderstandings.

Questions during rebuttal period:

Please address and clarify the cons above

Some typos:

Introduction, Notations, last line: a vector $\textbf{o}_{i} \in \mathbb{R}^{n} $ -> $\textbf{o} \in \mathbb{R}^{n} $
review_title: Official Blind Review #4 

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
+ This paper generalizes the supernet search problem on a broader horizon. Specifically, some of the current NAS methods use supernet to co-training different neural architectures for further architecture search. This paper does not just consider supernet as a tool for NAS, but also consider supernet as a graphical model and extend supernet to several general tasks in the form of graph data. (+)

+ This paper unifies the above tasks by a tensor formulation and encodes the topology inside the supernet by a tensor network. Different from NAS, the topological structure of the supernet is utilized in this paper. (+)

+ The paper further proposes an efficient algorithm that admits both stochastic and deterministic objectives to solve the search problem. (+)

+ A wide range of machine learning tasks besides computer vision are used to evaluate the proposed method's effectiveness, such as logic chain inference on knowledge graphs and meta-path discovery for HIN. (+)

- After reading the related work section, we notice that the authors have known some methods used in the NAS problem, but they are not aware of the current NAS's ineffectiveness problem. Specifically, the NAS's effectiveness is open to question, with its architecture rating is often inaccurate, especially in DARTS and some single-path one-shot NAS method. Two ICLR 2020 papers suggest that many NAS methods are not better than random architecture search. This paper borrows the concept from NAS but ignore the ineffectiveness/inefficiency of NAS. Therefore, the proposed method's effectiveness in this paper is questionable unless the authors provide an architecture rating analysis. (-)

In the context of chaotic phenomena in NAS nowadays, analyzing the architecture rating problem is of most importance. As there are many NAS papers published every year and their ineffectiveness may still be not widely recognized by the reviewers and the public, merely borrowing NAS methods to other domains (e.g., in this paper) is dangerous. This may cause more chaos in the new field. I think a NAS method's effectiveness should be sufficiently investigated before NAS can be generalized to a new domain.


- As defined, Ti in [0, 1] represents how ""good"" P can be. Then, there must be a connection between T and M(f(w, P), D), which represents the performance of learning model f(w, P) on dataset D? My understanding is that the authors can use the CORRELATION BETWEEN EDGE IMPORTANCE AND MODEL ACCURACY in the following paper [reference] to measure the connection between T and M. In other words, as is formulated by the authors, "Ti ∈ [0, 1] represent how "good" P can be," can the authors provide pieces of evidence showing the proposed method can actually have this effect? This is exactly related to the inaccurate architecture rating problem in NAS. Only can the authors answer this question will I be able to consider an acceptance of this paper. (-)


[reference]
@inproceedings{
anonymous2021dots,
title={{\{}DOTS{\}}: Decoupling Operation and Topology in Differentiable Architecture Search},
author={Anonymous},
booktitle={Submitted to International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=y6IlNbrKcwG},
note={under review}
}


+ The way to use routine (Algorithm 1) to tensorize the supernet is reasonable and correct. （+）


- This paper is very difficult to understand. I must read very carefully (especially for the subscript notation) twice until I can understand the paper. I think the main idea of this paper is simple. But the authors use a complicated way to describe the method, which makes the article unreadable and thus mysterious. (-)

Overall, this paper proposes a method that generalizes the idea from NAS to general tasks. However, it is open to question whether this generalization is appropriate as the supernet-based NAS's effectiveness is an active research topic.

review_title: A method that generalizes the idea from NAS to general tasks

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tries to use tensorize the supernet. The supernet itself is a graph and NAS aims to search the operations on the edges. The 'TRACE' finds that path-discovery appears in a number of tasks and NAS is one of them. 

This is an interesting topic and the experiments are conducted on vision/logic chain/node classification tasks. 

+) This paper is clear and easy to follow.

+) The topic is interesting. It is charming to design a universal method that could solve a number of problems.

However, there are still some concerns:

-) Is the title appropriate? The gradient-based NAS finds an optimal path in the global search space. However, for other tasks mentioned in this paper, is 'architecture search' appropriate to describe them? Could it be named as 'path search' or some others? Are there any other works support the 'architecture search' in other tasks?

-) The notations are not clear. The $\mathcal{J}$, the $r_{N_1(t)}$, the $\theta$ should all be described. However, I could not find them either in the 'annotation' paragraph or the first-mentioned sentences.

-) Move notations to the beginning of section 3.

-) What is $R_{N_1(t)}$ in Alg. 1? The $R$ has not been explained. However, I think this is important for the reader to understand the algorithm.

-) The results on CIFAR-10 aim to show that 'TRACE' could accelerate searching without loss of accuracy? What will happen if searching for longer epochs? According to 'Motivated by the success of previous methods (e.g., DARTS, SNAS and ENAS), our TN should include their encoding methods as special cases.', what is the number of parameters used for modeling the architectures? 

-) ' their decomposition method does not consider the topological structure of supernets,'. For the cell-based NAS in Table. 1. Why the topological structure has not been considered?

-) In Fig. 2, it's better to add a toy example based on Eq. 4 and add more explanations.

-) Please report the cost in Table 4, 5. For example, the params/search cost/flops in Table 3.

-) (minor) Lack of experiments on large-scale datasets, like ImageNet.

-) The HIN task searches for the rank-1 CP (Table 1), and the CV task also searches for the rank-1 solution. So this HIN task is just adopting the standard solution on another dataset? Are there any other differences?

-) What if the rank goes higher in Figure 3 (b)?

Based on the quality of this paper, I don't think this paper creates something new. In my opinion, this paper describes the solutions from a novel/universal perspective. Some of the existing methods are special cases. So I selected 5 as the initial score. It would be nice if the authors emphasize the unique contribution more clear and address the concerns. 
review_title: Review

Meta-review: This paper proposes a new and general formulation for supernet, which encodes supernet with tensor network(TN). The idea is interesting and motivated.  However, the paper is well presented and the clarify needs to be further improved.  The effectiveness of algorithm is not well justified and experimental results are less convincing even after additional results provided in the revision. Most importantly, it is not clear that the 'TENSORIZING' method can solve the current NAS's ineffectiveness problem.  It is confirmed that the reference to ICLR-2021 paper is not used for the decision of paper. 
Decision: Reject