Searching for Stage-wise Neural Graphs In the Limit
Authors: Xin Zhou
Dejing Dou
Boyang Li
Venue:  (2020)
Keywords: neural architecture search
graphon
random graphs

Abstract: Search space is a key consideration for neural architecture search. Recently, Xie et al. (2019a) found that randomly generated networks from the same distribution perform similarly, which suggest we should search for random graph distributions instead of graphs. We propose graphon as a new search space. A graphon is the limit of Cauchy sequence of graphs and a scale-free probabilistic distribution, from which graphs of different number of vertices can be drawn. This property enables us to perform NAS using fast, low-capacity models and scale the found models up when necessary. We develop an algorithm for NAS in the space of graphons and empirically demonstrate that it can find stage-wise graphs that outperform DenseNet and other baselines on ImageNet. 

Under review as a conference paper at ICLR 2020
SEARCHING FOR STAGE-WISE NEURAL GRAPHS IN THE LIMIT
Anonymous authors Paper under double-bling review
ABSTRACT
Search space is key consideration for neural architecture search. Recently, [32] found that randomly generated networks from the same distribution perform sim- ilarly, which suggests we should search for random graph distributions instead graphs. We propose graphon Cauchy sequence graphs and a scale-free probabilistic distribution, from which graphs of different number of nodes can be drawn. By utilizing properties of the graphon space and the associated cut-distance metric, we develop theoreti- cally motivated techniques that search for and scale up small caaallcc stage-wise graphs found on small datasets to large-capacity graphs that can handle ImageNet. The scaled stage-wise graphs outperform DenseNet and randomly wired Watts- Strogatz networks, indicating the benefits of graphon theory in NAS applications.
search space. A graphon is the limit of
INTRODUCTION
Neural architecture search (NAS) aims to automate the discovery neural architectures with high performance and low cost. Of primary concern to NAS is the design of the search space [23] which needs to balance multiple considerations. For instance, too small a space would exclude many good solutions, whereas space that is too large would be prohibitively expensive to search through. An ideal space should have a one-to-one mapping to solutions and sufficiently smooth in order to accelerate the search.
34 to keep the search space manageable is to search for small cell structure, typically containing about 10 operations with 1-2 input sources each. When needed, identical cells are stacked to form a large network. This technique allows cells found on, for instance, CIFAR-10 to work ImageNet. Though this practice is effective, it cannot be used to optimize the overall network structure.
common technique 37
17
35
19
24
In both manual and automatic network design, the overall network structure is commonly divided into several stages, where one stage operates on one spatial resolution and contains several near- identical layers multi-layer structures (i.e., cells). For example, ResNet-34 [1]] contains 4 stages with 6, 8, 12, and 6 convolutional layers, respectively. DenseNet-121 [12] contains 4 stages with 12, 24, and 16 two-layer cells. AmoebaNet-A (24] has 3 stages, within each 6 cells are arranged sequentially, Among cells in the same stage, most connections are sequential with skip connections occasionally used. As an exception, DenseNet introduces connections between every pairs of cells within the same stage.
Here we emphasize the difference between a stage and a cell. A cell typically contains about 10 op- erations, each taking input from 1-2 other operations. In comparison, stage can contain 60 or more operations organized repeated patterns and the connections can be arbitrary. network usually contains only 3-4 stages but many more cells. In this paper, we focus on the network organization at the level stage rather than cell.
32 recently showed that the stage structure can be sampled from probabilistic distributions of graphs, including Erdös-Rényi (ER) (1960), Watts Sttoggtt (WS) (1998), and Barabási-Ai Albert (BA) 11999,, yielding high- seghoreeforiingggg networks with low in-group variance. This finding suggests the random graph distribution, rather than the exact graph, is the main causal factor behind network performance. Thus, searching for the graph is likely not efficient searching for the random

---

Under review as a conference paper at ICLR 2020
(a) mo 10,
(b) mo 30,
(c) mo = 100,
100
n 300
1000
Figure 1: Three adjacency matrices of graphs generated by the Barabási-Albert model with m my 0.1n. black dot location (i. denotes edge from node i node j. The sequence of matrices converges to its limit, the graphon, as n + 00.
I-p
2-p
48
1-p
(a) Erdös Erdös-Rényi
(b) Watts- Strogati
Figure 2: Graphons for common random graph models. Different shades denote different proba- bilities (e.g.. p and 1 111pp p). The Erdös-Rényi model has two parameters: number of nodes and probability The Watts-Strogatz (WS) model has three parameters: number of nodes replace- ment probability p. and initial neighborhood width k. Technically, the WS model has a constant number of edges, violating exchangeability for random graphs; graphs sampled from (b) converges probability to the same number of edges as n increases.
graph distribution. The parameter space of random graph distributions may appear to be a good search space.
We propose a different search space, the space of graphons (20]. and argue for its superiority as an NAS search space. Formally introduced in Section a graphon is measurable function defined on [0, 1|2 - 00 1] and probabilistic distribution from which graphs can be drawn. Graphons are limit objects of Cauchy sequences of finite graphs under the distance metric.
Figure I visualizes three adjacency matrices randomly generated by the Barabási-Albert (BA) model with increasing numbers nodes. is easy to see that, as the number of nodes increases, the sequence of random graphs converges to its limit, graphon. The BA model starts with an initial seed graph with mo nodes and arbitrary interconnections. Here we choose complete graph as the seed. sequentially adds new nodes until there are n nodes in the graph. For every new node Unew- m edges are added, with the probability of adding an edge between Unew and the node v; being proportional to the degree of In Figure [ we let results in the same adjacency matrix suggests that directly searching in the parameter space will revisit the same configuration and is less efficient than searching in the graphon space.
mo 0.1n. The fact that different parameterization
Additionally, graphon provides a unified and more expressive space than common random graph models. Figure illlssrates the graphons for the WS and the ER models. We observe that these random models only capture a small proportion of all possible graphons. The graphon space allows new possibilities such interpolation striped combination of different random graph models.
Finally, graphon is scale-free, so we should be able to sample an aubitrary-sized stage-wise wise archi- tecture with identical layers (or cells) from graphon. This allows us perform expensive NAS
2

---

Under review as a conference paper at ICLR 2020
on small datasets (e.g., CIFAR-10) using low-capacity models and obtain large stage-war wise graphs to build large models. By relating graphon theory to NAS, we provide theoretically motivated tech- niques that scale up stage-wise graphs, which are shown to be effective in practice.
Our experiments aim to fairly compare the stage-wise graphs found by our method against DenseNet and the WS random graph model by keeping other network structures and other hyperparameters constant. The results indicate that the graphs found outperform the baselines consistently across a range of model capacities.
The contribution of this paper revolves around building a solid connection between theory and prac- tice. More specifically,
We propose graphon, generalization of random graphs, as a search space for stage-wise neural architecture that consists of connections among mostly identical units.
We develop an operationalization of the theory on graphon in the representation, scaling and search of neural stage-wise graphs that perform well in fair comparisons.
2 RELATED WORK
We review the NAS literature with a focus on the distinction between cell and stage structures. In the pioneering work of (36), recurrent neural network served as the controller that outputs all network parameters for all layers without such distinction. Later works attempted to reduce the cost of search by constraining the search to cell structures only. 35 searched for single cell structure and perform downsampling using pooling operations. (37.
33
and [24] searched for two types of
cells: a reduction cell that includes downsampling, and a normal cell that does not. [17] grew the cell structure from the simplest 1-operation cell to the maximum of 5 operations. DARTS [19] relaxed discrete choices and enabled gradient-based optimization. imposed probabilistic formulation on DARTS.
While various manual designs of stage structures have been proposed (e.g., Huang et al. 12 Larsson et al. (16). NAS for stage-wise graph has received relatively little attention. [32 found that random graphs generated by properly parameterized Watts-Strogatz models outperform manual designs for stage-wise structures. (30) redefined the stage-wise graphs that node take multiple inputs but output only a single channel, resulting in large stage-wise graphs with up to 2560 nodes. [27] evolved the connections between multiple residual blocks for applications in video processing. As a differennt type of global structure, [18 optimized organization of downsampling and upsampling. In summary, we believe that NAS for stage structures is an emerging research direction with many unexplored opportunities.
Another approach for accelerate search is to share weights among different architectures. 28 built a lattice where a chain-structuree network is path from the beginning to the end. In ENAS 22
controller is trained using gradient policy to select a subgraph, which is subsequently trained using cross-entropy. The process repeats, sharing network weights actoss training sessions and subgraphs. In the one-shot approach [4. (6) the hypergraph is only trained once. After that, subgraphs are sampled from the hypergraph and evaluated. Finally, the best performing subgraph is retrained. Our focus in this paper is to validate that the mathematical theory of graphon can be effectively operationalized and leave weight sharing future work.
BACKGROUND ON GRAPHON
Definition 1. A graphon is a symmetric, measurable function W [0, + [0.
The definition of graphon is straightforward, but its relation with graphs requires some standard concepts from real analysis, which we introduce below. In machine learning, graphon has found applications in hierarchical clustering g and graph elassification [10]
3

---

Under review as a conference paper at ICLR 2020
3.1 PRELIMINARIES
A metric space is space with a distance function. More specifically, a metric space is a set M with a metric function d : M x M - R that defines the distance between any two points in M and satisfies the following propertie: (1) Identity: d(r, r) = O. (2) Symmetry: d(æ, y) = d(x, y). (3) Triangular inequality: d(x,y) v) S d(x, 2) d(z, : As an example, the set of real numbers with y/ is metric space.
the absolute difference metric dabs (x, y)
Cauchy sequence defined as a sequence (#1 .2,,..) whose elements become infinitely close to each other as we move along the sequence. Formally, for any positive e E R+, there exists positive integer E 2 such that d(z,, Xi) < 6, Vi,j > N,i,j &
A complete metric space is metric space (M,d) in which every Cauchy sequence converges to point in M. It turns out that some familiar spaces, such as rational numbers Q under the metric dabe- not complete. As example, the sequence defined by xo 1, Zn-1 xn/2 + 1/2z, converges to irrational number V2. The space of real numbers R with dabe: however, is complete metric space. Given any metric space, we can form its completion, in general, by adding limit points. In the case of graphs, the limits allow explicit descriptions.
3.2 GRAPHON, THE CUT DISTANCE, AND DIGRAPHOO
We consider a weighted undirected graph (V, E) where every e V is associated with node weight a; and every edge eij E E is associated with edge weight 8ij. When necessary, we also write a; = a (G) to highlight the graph the node belongs to. The weighted graph is a generalization of the simple graph, where all nodes have weight and all edge have weight 1. Edges that do not exist are considered to have weight
3 show the following.
Theorem 1. Every Cauchy sequence of weighted graphs in the metric åy converges to a graphon.
Theorem 2. An weighted graphon is the limit of some Cauchy sequence in the metric &
The definition of the cut distance 8D relies on its discrete versions du and 8g. To avoid unnecessary technical details, here we introduce the intuition of dg and leave ór and 6g to Appendix
For partition S, T of V, the cut size is defined as cut(S, T,, 9) = Ee,es, O.ay When two graphs G and g' = (V", E') have the same set of nodes (i.e. V V'), the partition S, applies to both graphs. We can then define
d=(g,9')=
max S,TCV V
cut ( T,9) - (u, T, g')l
(1)
The seemingly counter-intuitive du is sensible metric for random graphs. Consider two random graphs with n nodes from the same Erdös-Rényi model with edge density 1/2. As they are iden- tically distributed, their distance should be small. However, their edit distance, or the number of edges where the two graphs differ, is likely quite large. In contrast, dn is only O(1/n) with high probability, which is consistent with our intuition.
For directed graphs with the potential of self-loops, 7 define digraphon as a 5-tuple (Woo, Wo1. W10. W11, w)). For nodes u, and Uj, (i. describes the probability that no edge exists between them; Wor (i,j) the probability that an edge goes from u; to uj; Wio(i, the proba- bility that an edge goes from Uj to u, and W (i,j) the probability that two edges go both ways. w; is the probability for a self-loop at u, For all i, j. Woo j) Wor (1 Wao () + (i.)).) = 1.
4 APPROACH
Since the graphon is a limit object, we must approaximate it with finite means. Here we use a step function approximation. We utilize a matrix B E [0, 1] n as the adjacency matrix of weighted graph whose edgeweights B. represent the mean of x square from the graphon. Its nodeweights uniformly 1 This approximation converges to the graphon when n tends to in- finity (Lemma 3.2, Borgs et al. (3). In order to represent directed acyclic graphs in neural networks, require the adjacency matrices to be upper-triangular and has zero vector the diagonal. This
4

---

Under review as a conference paper at ICLR 2020
is equivalent to imposing total ordering on the nodes and requiring for directed edge to go from U/ to V/y.
In the following, we propose theoretically motivated techniques for sampling stage-wise graphs of different sizes from the graphon representation and NAS for graphon.
4.1 SAMPLING FROM AND SCALING A GRAPHON
Given a finite graph Best with edge weights 3iy- E [0. 1] and uniform nodeweights, we can sample a simple graph Cummpe (i.c. with all edgeweights equal to 0 or 1) with nodes by drawing every edge independently 0-1 variable from the Bernoulli distribution parameterized by Biz. show ssampte converges to Gest in probability as the number of nodes increases.
Pr (d Grampte)
> 1 - 2
(2)
This procedure requires Gest and Gsample to have the same number of nodes.
By utilizing properties of the graphon metric space, we can sample graph with more than n nodes from the graph Gest with nodes. When the upsampling factor an integer k, we first create a new weighted graph Gest [k] with km nodes using the so-called k-fold blow-up procedure, and then use the above procedure to sample Geample with kn nodes. The k-fold blow-up of a graph Gest is created by splitting every node in Gest into k 1 nodes and connect the new nodes if and only if the original nodes are connected. More formally, for any two nodes i and Uj, we create copies Va..... Uik and U....... Uyk. The new graph Gese(k) [A] contains the edge from Usp to fig if and only there is an edge from Us to vy in the original graph. This scheme is justified as the cut distance
between graph and its k-fold blow-up G[k] is zero (See Corollary in Appendex D
(GG99[k)] = 0
(3)
To handle the case when the upsampling factor not an integer, we propose method called frac- tional blow-up. Suppose we want to create stage-wise wise graph with kn + m nodes. We first perform k-fold blow-up to create a new graph with kn nodes and equal nodeweight 1/kn. After that, we shift the nodeweights such that the first nodes have nodeweights 2/kkn+ m) and the rest kn m nodes have nodeweights 1/(kn + m). We subsequently split the first m node into 2 nodes, yielding graph of equal nodeweights. As detailed in Appendix D the cut distance between Gest and G can be bounded by
(kn - m)m
(Gese Pest) BB
kn(kn m)
where BA denotes the maximum difference between any edge weights. Since Big [0, 1], 3A < 1. From Gesto we can then sample a simple graph with kn + m nodes.
It is worth noting that the proposed upscaling methods differ from conventional upsampling tech- niques like linear or bilinear interpolation. In Appendix [D3] we show that, under moderate con- ditions, the k-fold blow-up graph is strictly closer the original graph than graph created by interpolation.
4.2 SEARCHING FOR AN ADJACENCY MATRIX
We now introduce the search algorithm. When optimizing discrete objects with gradient descent, the Gumbel softmax [14] 31) has been widely used. Given a multinomial distribution with proba- bilities To, RK. K we independently draw a K -ddimensional vector 3 from a Gumbel distribution: vk, 7h Gumbel(O, 1). The Gumbel softmax is defined as
exp (log TH + 71)r-1)
0g
(5)
exp (log n, + 71)
The Gumbel softmax can be understood as pitting the choices from 1 to against each other, while the random perturbation from enables exploration.
Our search algorithm optimizes the input connections to each node in the stage-wist wise graph. Em- pirically, we find it important to let different cells in the same stage learn to collaborate during the
5

---

Under review as a conference paper at ICLR 2020
node on the 0-1 lattice state on the lactiee visited by SGD the estimated graphon
Figure 3: The intuition on the search for the optimal graphon. When searching in the space of adjacency matrices, only can only move on the 0-1 lattice. The optimal graphon, denoted by the filled black dot, estimated by taking the average of the states visited by SGD.
search. Therefore, we devise an algorithm that pits different input combinations against each other. Specifically, the node v may take imput from nodes 11 44-1, from which we sample K subsets. For example, for the seventh node v7, we could sample {vi, V3 U5 or {ur, 26} and so on. We assign structural parameter mi every input subset. Finding the adjacency matrix amounts finding the best input subset for every node. We also assign separate model parameters to the same node in different input subsets. This allows all nodes the same subset learn to collaborate, which would be difficult if the parameters were shared across input subsets. The outputs of nodes in the same subset are aggregated using concatenation with zero padding for aligning the dimensions. During the search, the input to node 2 is computed the convex combination of the outputs from all K subsets, > K
ag (k). For the pseudo-code of the search algorithm, the reader is referred to
Appendix
We cross-entropy loss and standard stochastic gradient descent (SGD) with momentum opti- mize the model parameters together with subset weights л. After the search completes, We pick the input subset with the highest Ty the final adjacency matrix.
However, the goal of the search is to find probabilistic distribution of random graphs, not a sin- gle graph. As indicated by 21 and [3]. the last phase of SGD may considered as Markov chain that draws samples from stationary distribution. Figure SouureJilluse iillustrate this intuition. Thus, we compute the graphon as the average of adjacency matrix found by the last phase of the search.
5
EXPERIMENTS
5.1
SETUP
DenseNet [12] is classical example where the stage-wise structure plays major role in improving the network performance. Therefore, in the experiment, we focus on improving and comparing with DenseNet. In order create fair comparisons, focus the search the stage-wise structure and strictly follow DenseNet for the global network layout and the cell structure (See details in Appendix A.I.
Following 32 we start the search from the graphon that corresponds to the WS distribution with k/n 0.4 and p- 0.75, which denote as WS-G(0.4, 0.75). We limit the search input subsets that differ by edge from the starting point. We perform the search on CIFAR- 10 for 300 epochs with 8 cells every stage. The search completes within 24 hours on GPU.
We create four groups of comparison for the four DenseNet variants: DenseNet- 121, DenseNet- 169, 2e1, and DenseNet-264 in an increasing order of model capacity. We use the proposed scaling method (Section 4.1) to scale up the graphs in the four stages to roughly match the cor- responding DenseNet variant. The largest stage-wise graph, containing 64 nodes, used in the DenseNet-264 group. We also adjust the growth rate that the numbers of parameters of all models are as close to DenseNet as possible to enable fair comparisons. However, as the number of parameters depends distribution, we do not have precise control over the number parameters. We strictly follow the standard hyperparameters and data augmentation techniques used by DenseNet. The networks
the stage-wise connections, which are randomly drawn from the same

---

Under review as a conference paper at ICLR 2020
trained for 90 epoch on ImageNet; results from 6 independent training sessions averaged. For detailed list of hyperparameters and data augmentation, see Appendix A.2
We introduce two other baseline models besides DenseNet. In the first model, the stage-wise graphs are generated by randomly deleting edges from the fully connected graph of DenseNet. In the second model, we random graphs generated from WS-G(0.4, 0.75), which similar to the best random and DARTS 19
architecture in [32] and shown to be competitive with Amoeba [24] PNAS 17 on an equal-parameter basis.
5.2 RESULTS
For evaluation, we report the average performance on the ILSVRC-2012 validation set [26 and the new ImageNet V2 test set (25). For ILSVRR-2002 validation, we report the performance directly after epoch 90. For ImageNet V2, we select the best performing model on the ILSVRCC-012 vali- dation set from the 90 epochs and test on ImageNet V2. In order to mitigate the effects random variance, we sample 6 graphs from every graphon and report the average accuracy and standard deviation, as well as the number of parameters in Table T
Across all groups of comparison and model parameter setting, the graphon found by our algorithm consistently achieves the highest accuracy despite having the least parameters. On ImageNet vali- dation, the top-1 performance gap between our method and DensetNet is 0.4% except for 0.1 1%% for DenseNet-122 121. On ImageNet V2, the performance gap is up to 0.8%. The WS-G baseline in most comparisons is stronger than DenseNet but weaker than the propose technique.
5.3 DISCUSSION
We attribute the performance differences to the stage-wise graphs, since we have strictly applied the same setting, including the global network structure, the cell structure, and hyperparameter settings.
The first conclusion we draw is the effectiveness of the theoretically motivated scaling technique for graphon. We scaled up the 1-ndd graph found by the search to graphs with up 64 nodes the experiments. We also scaled the WS(4, 0.25) network, initially defined for 32 nodes in [32]. to 64 nodes in the DenseNet-264 group. The experiments show that after scaling, the relative rankings of these methods are maintained, suggesting that the proposed scaling technique incurs no performance loss.
Second, observe the standard deviations for most methods low, though they edge bit higher for ImageNet V2 where model selection has been carried out. This consistent with the findings of (32 and reaffirms that searching for random graphs is valid approach for NAS.
Finally, we emphasize that these results are created for the purpose of fair comparisons and not for showcasing the best possible performance. Our goal is to show that the graphon space and the associated cut distance metric provide a feasible approach for NAS and the empirical evidences support our argument.
6 CONCLUSIONS
The design of search space is of paramount importance for neural architecture search. Recent work 32 suggests that searching for random graph distributions is an effective strategy for the organi- zation of layers within one stage. Inspired by mathematical theories on graph limits, we propose new search space based on graphons, which are the limits of Cauchy sequences of graphs based on the cut distance metric.
The contribution of this paper is the operationalization of the graphon theory as practical NAS solutions. First, we intuitively explain why graphon superior search space than the parameter space of random graph models such as the Erdōs- Rényi model. Furthermore, propose technique for scaling up random graphs found by NAS to arbitrary size and present a theoretical analysis under the cut distance metric associated with graphon. Finally, describe operational algorithm that finds stage-wise graphs that outperform manually designed DenseNet as well as randomly wired architectures in [32

---

Under review as a conference paper at ICLR 2020
Table Performance (accuracy and standard deviation) on ImageNet and ImageNet V2.
Model
# Param
ImageNet Val
ImageNet V2 Test
Top-1
Top-5
Top-1
Top-5
DenseNet- 121
7.98 M
75.39 = 0.07
92.51 + 0.05
62.92 + 0.06
84.36 + 0.03
Random Deletion
8.02 M
75.30 = 0.09
92.48 = 0.07
63.41 0.31
84.74 = 0.13
WS-G (4, 0.75)
8.04
75.43 + 0.11
92.54 + 0.05
63.62 0.18
84.56 0.11
NAS Graphon
7.94
75.47 1: 0.12
92.57 0.04
63.74 0.08
84.71 0.24
DenseNet-169
14.15M
76.74+ 0.02
93.28 + 0.06
65.29+10.29
85.90++0221
WS-G (4, 0.75)
14.23M
76.94.10.06
93.37.10.07
65.18
85.79110.13
NAS Graphon
14.14M
77.14:0.16
93.44=0.099
65.36:10.25
85.96:10.14
DenseNet-201
20.0IM
77.42.10.06
93.60
65.84.1.117
86.04.10.09
WS-G (4, 0.75)
20.10M
77.56:0.111
93.67=0.077
65.99:10.09
86.29:10.15
NAS Graphon
19.94M
77.82=#0114
93.80=00009
66.44=0.15
86.59=00.38
DenseNet-264
33.34M
77.99:0.002
93.76=10.07
66.11:0.11
86.35:10.32
WS-G (4, 0.75)
33.42M
78.13=#0.13
93.93=#0.09
66.82=#0.28
86.54=0.24
NAS Graphon
33.26M
78.34++0.07
94.06+00.04
66.91+0.11
86.76+-0.1 14
Although find neural architectures with good performance, remind the reader that absolute performance is not the goal of this paper. Future work involves expanding the work to different operators in the same stage graph. This can be achieved, for example, the same manner that digraphon accommodates different types of connections. We contend that the results achieved in this paper should not be considered an upper bound, but only the beginning. of what can be achieved. We believe this work opens the door toward advanced NAS algorithms in the space of graphon and cut distance metric.
8

---

Under review as a conference paper at ICLR 2020
A IMPLEMENTATION DETAILS
A.1 THE GLOBAL STRUCTURE AND THE CELL STRUCTURE OF DENSENET
The DenseNet network contains a stem network before the first stage, which contains 3 x 3 convolution, batch normalization, ReLU and max-pooling. This is followed by three stages for CIFAR-10 and four stages for ImageNet. Between every two stages, there is a transition block containing a x 1 convolution for channel reduction and 2 2 average pool with stride 2 for downsampling. The network ends with a 7 x 7 global average pooling and a linear layer before softmax.
Channel Concat
BatchNorm
ReLU
Figure 4 shows the cell structure for DenseNet, which contains two convolutions with different kernel size: 1 and 3. Each of the two convolutions are immediately preceded by batch normal- ization and ReL.U. Every cell in the same stage outputs e channels. The input to the nhth cell is the concatenation of outputs from the cell to cell n - 1, for a total of c(n - 1) channels. As every cell in- crements the number of input channels by c, it is called the growth rate.
1x1 Conv
[channeks:
4c]
BatchNorm
ReLU
3x3 Conv
A.2 HYPERPARAMETERS
(chamnoss 4c o)
Some detailed hyperparameters in our experiments are as follows.
Figure 4: The DenseNet cell structure used in all experi- ments.
During the architecture search stage, train for 300 epochs. The learning rate schedule follows DenseNet, i.e. The initial learning rate is set to 0.1, and is divided by 10 at epochs 150 and 225. Mo- mentum is set at 0.9. Our batch size is 64 and growth rate in the DenseNet framework is set to 32. For the Gumble softmax, initial temperature is set at 1.0 and the minimum temperature is set at 0.1 with anneal rate of 0.03.
For ImageNet training, we train for 90 epochs. We use label smoothing, which assigns probability 0.1 all labels other than the ground truth. We use batch size of 256 64 per GPU GPUs or 260 = 52 per GPU x GPUs, depending on the GPU memory size. For the 4 stages, growth rates match the number of parameters for Densenet-121. Learning rate set at 0.1, divided by 10 at epochs 30, 60, and 80. We use Nesterov momentum of 0.9 and weight decay of 0.00005.
are set at 26,26.26.32
A.3 DATA AUGMENTATION
We also adopt the following data augmentation for ImageNet training, which are executed sequen- tially: Inception-style random cropping and aspect ratio distortion, resizing the image to 224 x 224, color jitter with factors for brightness, contrast and saturation all set to 0.4, horizontal flip with probability 0.5, and AlexNet-style random changes to the brightness of the RGB channels [15]
B THE SEARCH ALGORRTHM
Algorithm sshows the detailed procedures. Specifically, for every node v on the graph, we sample K subsets of nodes that could provide input to v, while avoiding cycles (lines 5-7). For example, for the seventh node U7, we could sample {11, Dy Us} or {rg. Ug} and so on. We assign one weight parameter тą to every input subset (line 8). The model parameters for node is denoted by 0,.. Every sampled input edge subset U(v, k) employ the same neural network operations in u, but with different parameters O,,k (lines 9-11). In particular, the input node the convex combination
9

---

Under review as a conference paper at ICLR 2020
Algorithm Search for Graphon
1: procedure INITIALIZE(V, k)
2: Input: totally ordered nodes V, number of subsets
3: Output: I (v), u, Ctu: D,
4: for each node 2 do
5:
I(v) {u u v. V)
all nodes preceding avoid cycles D the powerset of I(w))
6:
(P(u) < P(T(w))
7:
(+)) <- random K-element subset of (P()) Ot < a random vector in R
8:
weights for the subsets
9:
for each input subset U(v, k)
I.(v) do
10:
for each node E U(v, ks) do INITWEIGHT
create set of weights f,, for Output: feature map extracted by the current stage in(Via) <-
12: procedure STAGEFORWARD(Y, æ,, T, (t) O..i Te. U;)
Input: nodes V, input the stage æ, temperature т, initialized I (v), a n.. W;
14:
& Un is dummy sending outputs to nodes.
16:
for each node E V do for each input subset U(v, k) IL((v) do for each node E U(v, k) do out(u, k) + (in(()), 0u,k)
17:
18:
19:
> apply the operation of u to its input concatenation
20:
out(k) AGGREGATH Vu)
Sample y Gumbel(O,
22: Qę + exp((log n.. +11//7)
>> perform Gumbel softmax
Σ, expillog F.., +T///)
(n) EK ag out(k)
return in (Vout)
>> Vout is dummy whose input is the stage's output.
of the outputs from all K subsets.
K in(u)= = Σ aj out(k)
(6)
This allows all nodes in the same subset to learn collaborate, which would be difficult if the parameters were shared across input subsets. During the forward computation, outputs from nodes in the same input subset different input subsets are forced to compete by the Gumbel softmax (lines 21-23). We pick the input edge subset with the highest ग as the winning adjacency matrix.
aggregated by either summation concatenation (line 20). Finally,
C
THE CUT DISTANCE &
For partition S, T of V, the cut size is defined as
(St T. 9)
Σ
0,
When graphs G and g' (V', "V),E have the applies to both graphs. We can then define
of nodes (i.e. V V"), the partition S, T
d=(G,9')= = max
cut (S, (,,,,,TT,, T,9) - cut(S, T,G')
(8)
S.TC V
When the graphs have the same number of nodes, but the correspondence between nodes is un- known, the distance is defined as the minimum over all isomorphism ς ~ G.
(n = min max d_(9.9))
(9)
Gegg STTC
10

---

Under review as a conference paper at ICLR 2020
In general, the two graphs do not have the same number of nodes. Thus, in the most general metric on, we allow the correspondence between nodes to be fractional. We define the "overlay" matrix E IR MRXX", The entry Lip denotes the fractional mapping from u, E V to up E V subject to
B
M
Σ
Lip = (g) and Σ Lip - a;(9)"
(10)
2m
Based on L, we can create two new graphs, G[L)] and G. which have the same set of nodes V x V and well defined distance 9' is Lip- The edge weights are different, however; the weight of edge ((u,, Up), (u,, uy)) )) in G is
In both graphs, the weight of node (i4 up)
Bij. whereas in [L is 8pg. The cut distance du takes the minimum over all possible overlay matrices.
(") min
(414))
(11)
LEC
The fractional overlay is applicable even when the two graphs have the same node count and can lead to lower distance than integer overlay.
D FRACTIONAL. UPSAMPLING
In fractional upsampling, start with graph 9 with n nodes and generate new graph with + m nodes (0 < m < n). To achieve this, we perform two operations sequentially. First, we shift the node weights such that the m nodes have weight 2/(n + m) and the rest n m nodes have weights 1/(n + m.) Next, in what we call partial blow-up, we split each of the m nodes into two nodes. After the upsampling operation, all nodes have equal weights 1/(n + m). The requirement for equal weight is needed as the nodeweights are the probabilities of every node being sampled.
Here, the w-w split of node v; defined as replacing u with k new nodes Ui...... Usk, each having node weight a./k. k. For any other node Vy the edge (Vax, 0j) has the same weight as (V;, U2) and the same applies to (vj, U.x).
In the following, we analyze the cut distance between the new graph and the original graph and show its upper bound is (u-m)m BA, where BA denotes the maximum difference between any two edgeweights.
m[n+m]
D.1 WEIGHT SHIFTING
Theorem 3. Let G = (V,B) and G' -V, be two weighted graphs thut have the same set of n nodes with different nodeweights and the same set of edges with the same edgeweights. If every node in & has nodeweight 1/n, whereas G' has nodes (0 < < n) with nodeweight 2/(n + m) and n - m nodes having weight 1/(n + m). then
(n m m)m
(9, S BB
(12)
n(n + m))
where Ba denotes the maximum difference between any two edge weights.
Proof. We create an overlay matrix L with the diagonal terms L;: mii((a(()) agG"). For the first m diagonal terms, L = 1/n, and for the rest the diagonal terms are 1/(n + m). Then
(G < ((G S
Σ
LipLio (3: - Bpg (G)).
(13)
F==nh and = q, then GG) = (G). Thus,
Σ 4..P.9
Tai (8kg (G) - 3pp(G)) = B3
Σ
Lip
isp or jott
2 5)
(14)
Ba (n- m)m
BA
1
Σ Liss
n(n + m)
11

---

Under review as a conference paper at ICLR 2020
D.2 PARTIAL BLOW-UP
Theorem 4. Let G EV be a weighted graph without self-loops and G' be the graph resulted from the k-way split of a node v; E V, (, g")
Proof. We construct the overlay matrix L as follows. Laj = 1/n, Vj + i. Lip = 1/kn, for p -,.... k. All other entires in L are zeros.
Thus, the graphs G[L] and G'[LT] have the same set of nodes with non-zero weights, which include (ug; uz) for j o and (us; Uip) for p ....., k. The nodes (uj; have weight 1 and the nodes (4s, Upp) have weight 1/k. The rest of the nodes have weight O and do not enter the computation of dn-
In both 9[4] and J'[L"I, for all j.j' # i, the edge weight between (uj, uy) and (ulj uf) is the same weight in the original graph (g), Additionally, in both G[L] and G' the edge weight between (uj, uy).j F i and (u:, u = 1, 1,...,k , is (g) In [[L]. there are no self loops, so (9) = 0. Similarly, in g'LL there are no edges between Uip and Wig. so (i') = O. Therefore, the graphs G[L] and "] are identical and (
= O.
Since the metric dn is non non-negative,
9
minLec da(G) =.
Corollary 1. Let G (V, E) be weighted graph and G' be the graph resulted from the k-way split of any subset of nodes V. S V. then 6n(9, G') O.
Proof. g' can be realized by splitting the nodes in V. S V sequentially. Every split keeps the distance with the previous graph at zero.
Corollary 2. Let g = (V, E) and G[k] be a weighted graph and its k-way blow-up, respectively, then 6n (9.99KK))
D.3 COMPARING K-FOLD BLOW-UP WITH INTERPOLATION
Another possible approach for upsampling of a weighted graph is interpolating its adjacency ma- trix. We analyze the distance between the interpolated graph w.r.t. the blow-up graph and show interpolation results in higher distance under moderate assumptions.
In this analysis, we adopt the following notations and definitions. Let G denote weighted graph with n nodes. The adjacency matrix of G is upper-triangular and has an all-zero diagonal. This is the same as the graphon parameterization that we adopt in this paper. Further, let [k] be the k-fold blow-up of G and let [k)] be a graph obtained by interpolating the adjacency matrix of G to kn nodes. In all three graphs, let the sum of nodeweights be and evenly distributed among all nodes.
We focus on the analysis of the one-dimensional linear interpolation, where Big (G [k])) (i/k: [
L5/k + (Ti/k] T// (9). For example, if in- terpolate a 3 x 3 matrix to the size 3k x 3k. we would first repeat every row k times and interpolate horizontally. More specifically, the row [a, (a,b, c] will become Ja, (k-1)a+b .....
a+(k-116 [K- 11b+t b+(k-1)e [k-1)e) []. We note the row sum is k
k+1 2
kb + kc.
Theorem S. (I) the adjacency matrix of 9 contains column with non-zero sum and (2) girl [K] and G [i] are optimally overlaid (i.e. 8 (g (a)l s (" = (k)) then
80 (9, (k)) (()))
Proof. From Theorem 2.3 of 3, we have the inequality
67
)
< En (G]K), gi)l
(15)
32
12

---

Under review as a conference paper at ICLR 2020
Note that Glk and [[itp](k)] have equal number of nodes. By definition, for partition 8, T of the node set V.
8 (g [1]
(S, T, ([k]) - (S, V, (k)*)
max 8,T
max
Σ
(16)
S,T
Σ (B,((TT)) (e)(T()()))) R.ER
max S,T 2
Now we construct the partition S', T' such that the first km nodes belong to S' and the rest kn - km nodes belong to T'. Since the adjacency matrix of G upper triangular and non-zero, without loss mt of generality we pick such that Σi=1 Bim (G) K Σx=++ Bam (9). Since all entries below the
diagonal are zero, that inequality is satisfied as long as the column sum at the index m is nen-zero. Hence, let S' {(r. Ukm and T {lkm+1,.... Ukn). Let the cut distance under this partition S', T be 5 (uol)) then
("Pl
1
Σ
(BG(9[k)) Beg (gr
k2
M. ET
B9 + Σ
Bi
2
j=m++1
imm
km
kn
kn
km
Biy (k))
Big (k))
(17)
j=m++1
i=km
m
3im (9)
Σ
Bim + Σ B((())
222
2
imm+1
1mm
3im (9)
Σ
Bim (9)
> 0
2
2
i=m+1
Thus, ån
2
O. By Eq 15 dn (G[A),
> 0, implying
dn (u)))) > O. Since 8 [k)) = O. we have proven the desired proposition.
D.4 DISCUSSION
Theorem 5 shows the k-fold blow-up method better approximation of the original graph in terms of the cut distance &n than the ID linear interpolation. But the exact k-fold blow-up is only applicable when & an integer. If a graph of size n +0 < m < n)) is desired, we need to resort to the fractional blow-up method, which has been analyzed in Theorems 3 and 4 We show that when m is or n - 1, this partial blowup operation does not cause 8 to change more than O(3a/n) However, when is n/2, 8n between the original graph and the new graph could be up to 3a /6. This suggests that the fractional upsampling results in a graph that is similar to the original when only small number of nodes (relative to is added.
REFERENCES
[1] Albert-Lász16 Barabási and Réka Albert. Emergence of scaling in random networks. 286: 609-512, 1999.
[2] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Under- standing and simplifying one-shot architecture search. In ICML. 2018
13

---

Under review as a conference paper at ICLR 2020
[3] Christian Borgs, Jennifer T Chayes, László Lovász, Vera T Sós, and Katalin Vesztergombi. Convergent sequences of dense graphs i: Subgraph frequencies, metric properties and testing. Advances in Mathematics, 21966):1011-18511 2008.
[4] Andrew Brock, Theodore Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shot model architecture search through hypernetworks. arXiv preprint arXiv:1708.05344, 2017.
[5] Francesco Paolo Casale, Jonathan Gordon, and Nicolo Fusi, Probabilisttic neural architecture search. CoRR, abs/1902.05116, 2019. URL
05116
[6] Yukang Chen, Tong Yang. Xiangyu Zhang, Gaofeng Meng, Chunhong Pan, and Jian Sun. Detnas: Neural architecture search on object detection. arXiv preprint arXiv:1903.
[7] P. Diaconis and S. Janson. Graph limits and exchangeable random graphs. Rendiconti di Matematica delle sue Applicazioni, 28(1):33 61, 2008
[8] Justin Eldridge, Mikhail Belkin, and Yusu Wang. Graphons, mergeons, and so on! In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Gamett (eds.), Advances in Neural Infor- mation Processing Systems 29, pp. 2307 2315. Curran Associates, Inc., 2016. URL http: and- so-on--s-on... pdf
7/papers wttpppppccppppppppepppepeeppep 6089 garggpppoooog--appoooososoo
[9] P. Erdós and A Rényi On the evolution of random graphs. In Publication of the Mathematical Institute of the Hungarian Academy of Sciences, pp. 17-61, 1960.
[10] Andreas Haupt, Mohammad Khatami, Thomas Schultz, and Ngoc Mai Tran. Classification on large networks: A quantitative bound via motifs and graphons. arXiv Preprint, arXiv 1710.08878, 2017.
[11] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[12] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely nected convolutional networks. In CVPR. 2017.
[13] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads wider optima and better generalization. In UAI, 2018.
[14] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with Gumbel-softmax. 2017.
[15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. Commumunications of the ACM, 60(6):84-90 2017.
[16] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. In ICLR, 2017.
[17] Chenxi Liu, Barret Zoph. Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV. 2018.
[18] Chenxi Liu, Liang-Chieh Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei-Fei. Auto-Deepl ab: Hierarchical neural architecture search for semantic image seg- mentation. CVPR. 2019.
[19] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019.
[20] László Lovász and Balázs Szegedy. Limits of dense graph sequences. Technical Report Tech- nical Report TR- 2004770, Microsoft Research, 2004.
[21] Stephan Mandt, Matthew D. Hoffma, and David M. Blei. A variational analysis stochastic gradient algorithms. In ICML, 2016.
[22] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via via parameters sharing. In Proceedings of the 35th International Conference on Machine Learning, pp. 4095 2018.
14

---

Under review as a conference paper at ICLR 2020
[23] Ilija Radosavovic, Justin Johnson, Saining Xie, Wan- Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. In ICCV, 2019.
[24]
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image classifier architecture search. In AAAI. 2019.
[25] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Proceedings of the 36th International Conference on Machine Learning, pp. 5389-58880 -500,, 2019.
[26] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-22) 252, 2015. doi: 10.1007/s11223-015-000
[27] Michael S. Ryoo. AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenet: Searching for multi-stream neural connectivity in video architectures. arXiv Preprint. arXiv 1905. 13209, 2019.
[28] Shreyas Saxena and Jakob Verbeek. Convolutional neural fabrics. NIPS. 2016.
[29] D. J. Watts and S. H. Strogatz. Collective dynamics of "small-world" networks. 393:440-442, 1998.
[30] Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. 2019.
[31] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, pp. 10734-10742, 2019.
[32] Saining Xie, Alexander Kirillov, Ross B. Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. In ICCV, 2019.
[33] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. SNAS: Stochastic neural architecture search. In ICLR, 2019.
[34] Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas-bench- Towards reproducible neural architecture search. 2019.
[35] Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu. Practical block-wise neural network architecture generation. In CVPR. 2018.
[36] Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. In ICLR. 2017.
[37] Barret Zoph, Vijay Vasudevan, Jonathan Shlens, and Quoc V. Le. Learning transferable archi- tectures for scalable image recognition. CVPR, 2018.
15


Reviews:
[{"review_id": "BklbTSL0Fr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new graphon-based search space. Unlike most other NAS works that search for exact network structures, this paper aims to search for the random graph distribution with graphon. Overall, it provides some new angles for NAS search space design, but the experimental results are very weak.\n\n1.  It simply ignore all other NAS works and just compares with the baseline DenseNet and random deletion/walk (WS-G). Despite that, the gain (accuracy +0.17% than DenseNet baseline) is very marginal compared to other approaches:  random-wire (accuracy +2% than resent50 baseline), FBNet (accuracy +2% than MobileNetv2 baseline).\n2. According to Section 5.1, the search is performed on CIFAR-10, but there is no evaluation on CIFAR-10 at all. The only results are reported for ImageNet instead, which is kind of strange.\n\nGiven these weak results, I cannot accept this paper in the current form.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "B1xkzupttH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a new search space based on graphons and explore some of its benefits such as certain theoretical properties. The architecture search shares similarities with DARTS. An important difference is that the network parameters are not shared.\nThe paper is well-written and the authors consider that the typical reader will not be familiar with graphons. I agree that their proposed model allows for more architectures but in practice it is not much stronger than WS-G. The argumentation with respect to parameters is unclear to me. On one hand, you manually influence the number of parameters, on the other you argue that you use less parameters. Obviously, you chose that your baselines have more parameters. How do results for WS-G look like if you reduce its parameters to match yours? In fact, you were searching for an architecture on CIFAR-10 but you did not report your results here. Instead you only report your transferred results to ImageNet. Is it possible that you also report results on CIFAR-10? Finally, you do not discuss that your graph contains only one kind of node. In many NAS methods the search space contains various types of operations. Do you think this is a problem? Is there a trivial way to extend your method to cover this as well?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a graphon-based search space for neural architecture search. Unfortunately, the paper as currently stands and the small effect sizes in the experimental results raise questions about the merits of actually employing such a search space for the specific task of NAS. The reviewers expressed concerns that the results do not convincingly support graphon being a superior search space as claimed in the paper. 

Decision: Reject