Title: TASK LEVEL DATA AUGMENTATION FOR META-LEARNING Anonymous authors Paper under double-blind review

Abstract: Data augmentation is one of the most effective approaches for improving the accuracy of modern machine learning models, and it is also indispensable to train a deep model for meta-learning. However, most current data augmentation implementations applied in meta-learning are the same as those used in the conventional image classification. In this paper, we introduce a new data augmentation method for meta-learning, which is named as "Task Level Data Augmentation" (referred to Task Aug). The basic idea of Task Aug is to increase the number of image classes rather than the number of images in each class. In contrast, with a larger amount of classes, we can sample more diverse task instances during training. This allows us to train a deep network by meta-learning methods with little over-fitting. Experimental results show that our approach achieves state-of-the-art performance on miniImageNet, CIFAR-FS, and FC100 few-shot learning benchmarks. Once paper is accepted, we will provide the link to code.

1. INTRODUCTION:

Although the machine learning systems have achieved a human-level ability in many fields with a large amount of data, learning from a few examples is still a challenge for modern machine learning techniques. Recently, the machine learning community has paid significant attention to this problem, where few-shot learning is the common task for meta-learning (e.g., Ravi & Larochelle (2017) ; Finn et al. (2017) ; Vinyals et al. (2016) ; Snell et al. (2017) ). The purpose of few-shot learning is to learn to maximize generalization accuracy across different tasks with few training examples. In a classification application of the few-shot learning, tasks are generated by sampling from a conventional classification dataset; then, training samples are randomly selected from several classes in the classification dataset. In addition, a part of the examples is used as training examples and testing examples. Thus, a tiny learning task is formed by these examples. The meta-learning methods are applied to control the learning process of a base learner, so as to correctly classify on testing examples.

Data augmentation is widely used to improve the training of deep learning models. Usually, the data augmentation is regarded as an explicit form of regularization He et al. (2016) ; Simonyan & Zisserman (2014) ; Krizhevsky et al. (2012) . Thus, the data augmentation aims at artificially generating the training data by using various translations on existing data, such as: adding noises, cropping, flipping, rotation, translation, etc. The general idea of data augmentations is increasing the number of data by change data slightly to be different from original data, but the data still can be recognized by human. The new data involved in the classes are identical to the original data. However, the minimum units of meta-learning are the tasks rather than data. Increasing the data of original class cannot increase the types of task instances. Therefore, "Task Aug" increases the data that can be clearly recognized as the different classes as the original data. With novel classes, the more diverse task instances can be generated. This is important for the meta-learning, since metalearning models must predict unseen classes during the testing phase. Therefore, a larger number of classes is helpful for models to generate task instances with different classes.

In this work, the natural images are augmented by being rotated 90, 180, 270 degrees (we show examples in Figure 1 ). We compare two cases, 1) the new images are converted to the classes of original images and 2) the new images are separated to the new classes. The proposed method is evaluated by experiments with the state of art meta-learning Methods Snell et al. (2017) ; Lee (2017) . The experimental result analysis shows that Task Aug can reduce over-fitting and improve the performance, while the conventional data augmentation (referred to Data Aug) of rotation, which converts the novel data into the classes of original data, does not improve the performance and even causes the worse result. In the comparative experiments, Task Aug achieves the best accuracy of the meta-learning methods applied. Besides, the best results of our experiments exceed the current state-of-art result over a large margin.

2. RELATED WORK:

Meta-learning involves two hierarchies learning processes: low-level and high-level. The low-level learning process learns to deal with general tasks, often termed as the "inner loop"; and the highlevel learning process learns to improve the performance of a low-level task, often termed as the "outer loop". Since models are required to handle sensory data like images, deep learning methods are often applied for the "outer loop". However, the machine learning methods applied for the "inner loop" are very diverse. Based on different methods in the "inner loop Schwartz et al. (2018) . The later tried to apply meta-learning in a few-shot generation to generate samples from other models Antoniou et al. (2017) . In addition to these two types of studies, the data augmentation technology most closed to the new proposed approach is applied to Omniglot dataset, which consists of handwritten words Lake et al. (2015) . They created the novel classes by rotating the original images 90, 180 and 270 degrees Santoro et al. (2016) . However, this approach cannot be applied for the natural color image directly, and we will explain the reasons and the solutions in Section 3.

3. METHOD:



3.1. PROBLEM DEFINITION:

We adopt the formulation purposed by Vinyals et al. (2016) 

D tr = {(x k n , y k n )|n = 1 . . . N ; k = 1 .

. . K} consists of K data instances from each class, this processing usually called as a "shot". The validation set, D val , consists of several other data instances from the same classes, this processing is usually called as a "query". An evaluation is provided for generalization performance on the N classification task instance D tr . Note that: the validation set of a task instance D val (for optimizing model during "outer loop") is different from the held-out validation classes set C val and meta-set S val (for model selection).

3.2. CREATING CLASSES BY ROTATING CLASSES:

This work is to increase the size of the training classes set, M tr , by rotating all images within the training classes set with 90, 180, 270 degrees. The size, M tr , is increased for three times. In the Omniglot dataset consisting of handwritten words Santoro et al. (2016) , this approach works well, since it can rotate a handwritten word multiple of 90 degrees and treat the new one as another word; in addition, it is really possible that the novel word is similar to some words, which are not included in the training classes but existed. However, for natural images, it is not the same cause. For examples, the images in the third line of Figure 1 are difficult to identify which images are rotated. Moreover, the images are rarely rotated in the photos taken by humans.

Despite the two problems, the fundamental features of the novel images can provide useful information. We assign novel classes that contain smaller weights than the original classes, so as to make models prioritize learning the features of the original classes, and make the features of the novel classes as a supplement to prevent the augmented data from taking up large capacity in the model.

The smaller weights are implemented in two ways, 1) lower probability and 2) delay selecting the novel classes. For a class in a task instance, the probability of the class coming from the novel classes is p, and the probability coming from the original classes is 1 -p. Besides, The initial p is set to 0, then linearly rises from 0 to p max after generating T task instances. The max probability p max is set lower than the proportion of the novel classes in all classes to make each novel class have a lower probability than each original class. The whole process of Task Aug on a classes set is summarized in Algorithm 1 and Figure 2 .

3.3. ENSEMBLE:

In this work, we also compare the methods with the training protocol with ensemble method Huang et al. (2017) an ensemble model, in order to better use the models obtained in a single training process, and this approach has been proved to be valid for meta-learning by experiments Liu et al. (2018) . We adopt this ensemble method. However, unlike Huang et al. (2017) and Liu et al. (2018) that we did not use cyclic annealing for learning rate and any methods to select models. We directly took the average of the prediction of all models, which are saved according to an interval of 1 epoch. In Section 4, the methods with this ensemble approach are marked by "+ens".

4. EXPERIMENTS:

We evaluate the proposed method on few-shot learning tasks. In order to ensure fair, both the results of baseline and Task Aug were run in our own environment. The comparative experiment is designed to answer the following questions: (1) Is Task Aug able to improve the performance of meta-learning? (2) How much should the probably for the novel classes be set? (3) Will converting the novel data into the classes of the original data cause worse results, which are generated by being rotated 90, 180, 270 degrees? The block sizes of DropBlock were set to 2 and 5 for CIFAR derivatives and ImageNet derivatives, respectively. In all experiments, the dropout possibility was set to 0.1. The global average pooling was not used for the final output of the last residual block. For M-SVM, we set training shot to 5 for CIFAR-FS; 15 for FC100; and 15 for miniImageNet; regularization parameter of SVM was set to 0.1; and a learnable scale was used following Lee et al. (2019) . We did not use label smoothing like Lee et al. (2019) , because we did not find that label smoothing can improve the performance in our environment. This was also affirmed from the Lee et al. (2019) author's message on GitHub, that Program language packages and environment might affect results of the meta-learning method.

For R2-D2, we set the same training shot as for M-SVM, and used a learnable scale and bias following Bertinetto et al. (2018) . It was different from Bertinetto et al. (2018) we used a fixed regularization parameter of ridge regression which was set to 50 because Bertinetto et al. (2018) has confirmed that making it learnable might not be helpful.

Last, for all methods, each class in a task instance contained 6 test (query) examples during training and 15 test (query) examples during testing.

4.1.3. TRAINING CONFIGURATION:

Stochastic gradient descent (SGD) was used. Following Sutskever et al. (2013) , we set weight decay and Nesterov momentum to 0.0005 and 0.9, respectively. Each mini-batch contained 8 task instances. The meta-learning model was trained for 60 epochs, and 1000 mini-batchs for each epoch.

We set the initial learning rate to 0.1, then multiplied it by 0.06, 0.012, and 0.0024 at epochs 20, 40 and 50, respectively, as in Gidaris & Komodakis (2018) . The results, which are marked by "ens" were used the 60 models saved after each epoch to become an ensemble model. For data augmentation, we adopted random random crop, horizontal flip, and color (brightness, saturation, and contrast) jitter data augmentation following the work of Gidaris & Komodakis (2018) ; Qiao et al. (2018) . We set p max to 0.5 for CIFAR-FS and FC100; 0.25 for miniImageNet; and T was set to 80000 for all experiments. The FC100 Oreshkin et al. (2018) are also derived from CIFAR-100 Krizhevsky et al. (2010) , and the 100 classes are grouped into 20 superclasses. The training, validation, and testing classes contain 60 classes from 12 superclasses, 20 classes from 4 superclasses, and 20 classes from 4 superclasses, respectively. The target is to minimize the information overlap between classes to make it more challenging than current few-shot classification tasks. Same as CIFAR-FS, there are 600 nature color images of size 32 × 32 in each class.

Results.:

In Table 1 , we compare our results with the previous studies, and the table shows that the highest accuracies of our experiments exceeded the current state-of-art accuracies from 3% to 5%. Besides, Table 2 and Table 3 summarize the results on the CIFAR-FS and FC100 5-way tasks, and in most cases our method rises accuracy by 0.5%-3%. We can observe that: some results without the ensemble approach Huang et al. (2017) of baseline and Task Aug are close, but the advantage of Task Aug is still obvious on the results with the ensemble approach. We suspect that the scale of backbone limits the performance of the best model. A larger scale backbone is needed for the training process with Task Aug. For the results of ensemble approach, since Task Aug reduces the over-fitting, more models during the training process have good performance, which provide ensemble with models of higher quality. Results. Table 4 shows that the highest accuracies of our experiments exceeded the current stateof-art accuracies from 1% to 2%. Table 5 summarizes the results on the miniImageNet 5-way tasks, and in most cases our method rises accuracy by 0.5%-2%. Figure 3 : The accuracies (%) on meta-test sets with varying probability p max for the novel classes.The 95% confidence interval is denoted by the shaded region. In general, the performance of Task Aug on most of the regimes is better than Data Aug and baseline.

4.4. EFFICIENCY OF THE PROBABILITY FOR TASK AUG:

To identify whether the rotation multi 90 degrees for Task Aug is better than that for Data Aug, we analyzed the experiment on CIFAR-FS and miniImageNet. The linear rising of p was also used for Data Aug, and T = 80000 for both Task Aug and Data Aug. In the analysis, the training classes set was not augmented by the validation classes set. As shown in Figure 3 , we observed that: with p max , the accuracy rises at first, reaches the peaks between 0.25 and 0.5, then declines and reaches baseline when p max = 0.75 at the end, which is the proportion of the novel classes in all classes.

On the other hand, the rotation multi 90 degrees for Data Aug can not improve or even cause worse performance.

5. CONCLUSION:

We proposed a Task Level Data Augmentation (Task Aug), a data augmentation technique that increased the number of training classes to provide more diverse few-show task instances for metalearning. We proved that Task Aug was valid for CIFAR-FS, FC100, and miniImageNet, and exceeded the result of the previous works. Task Aug achieved the performance by rotating the images 90, 180 and 270 degrees. This method is simple and cost-effective. With the ensemble method, we exceeded the state-of-the-art result over a large margin.

Future work will focus on searching different network structures for meta-learning, since the training with Task Aug would require more large model. Besides, we will try to apply Task Aug to other fewshot learning tasks to verify its effectiveness. Another interesting topic is to build other approaches for Task Aug, such as swapping channel order, picture blend or even auto augmentation.

:



Figures and Tables:

Figure 1: Examples of the novel created classes.

Figure 2: The process of generating a task instance with Task Aug.

al. (2019);Oreshkin et al. (2018);Mishra et al. (2017), we used ResNet-12 network in our experiments. The ResNet-12 network had four residual blocks which contains three 3 × 3 convolution, batch normalization and Leaky ReLU with 0.1 negative slope. One 2 × 2 max-pooling layer is used for reducing the size of the feature map. The numbers of the network channels were 64, 160, 320 and 640, respectively. DropBlock regularizationGhiasi et al. (2018) is used in the last two residual blocks, the conventional dropoutHinton et al. (2012) is used in the first two residual blocks.

For the final epoch, the training classes set was augmented by the validation classes set. We chose the model at the epoch where we got the best model during training on the training classes set only. The results of the final run are marked by "+val" in this subsection.

EVALUATION ON MINIIMAGENETThe miniImageNetVinyals et al. (2016) is one of the most popular benchmark for few-shot classification, which contains 100 classes randomly selected from ILSVRC-2012Russakovsky et al. (2015). The classes are randomly divided into training classes, validation classes and test classes, and them contain 64, 16 and 20 classes, respectively. There are 600 nature color images of size 84 × 84 in each class. SinceVinyals et al. (2016) did not release the class splits, we use the more common split proposed byRavi & Larochelle (2017).

to describe the N -way K-shot task. A few-shot task contains many task instances (denoted by T i ), each instance is a classification problem consisting of the data sampled from N classes. The classes are randomly selected from a classes set. The classes set are split into M tr , M val and M test for a training class set C tr , a validation classes set C val , and a test classes set C test . In particular, each class cannot overlap others (i.e., the classes used during testing are unseen classes during training). Data is randomly sampled from C tr , C val and C test , so as to create task instances for training meta-set S tr , validation meta-set S val , and test meta-set S

Table content: <table/>

The average accuracies (%) with 95% confidence intervals. * CIFAR-FS results fromBertinetto et al. (2018). † Result fromLee et al. (2019). The best results are highlighted.

Table content: <table><tr><td>Method</td><td colspan="2">CIFAR-FS 5-way 1-shot 5-shot</td><td colspan="2">FC100 5-way 1-shot 5-shot</td></tr><tr><td>MAML  *  Finn et al. (2017)</td><td>58.9±1.9</td><td>71.5±1.0</td><td>-</td><td>-</td></tr><tr><td>R2-D2 Bertinetto et al. (2018)</td><td>65.3±0.2</td><td>79.4±0.1</td><td>-</td><td>-</td></tr><tr><td>TADAM Oreshkin et al. (2018)</td><td>-</td><td>-</td><td>40.1±0.4</td><td>56.1±0.4</td></tr><tr><td>ProtoNets  † Snell et al. (2017)</td><td>72.2±0.7</td><td>83.5±0.5</td><td>37.5±0.6</td><td>52.5±0.6</td></tr><tr><td>MTL Sun et al. (2019)</td><td>-</td><td>-</td><td>45.1±1.8</td><td>57.6±0.9</td></tr><tr><td>M-SVM Lee et al. (2019)</td><td>72.8±0.7</td><td>85.0±0.5</td><td>47.2±0.6</td><td>62.5±0.6</td></tr><tr><td>M-SVM (+ens+val) (our)</td><td colspan="4">76.75±0.46 88.38±0.33 49.83±0.45 67.14±0.42</td></tr><tr><td>R2-D2 (+ens+val) (our)</td><td colspan="4">77.66±0.46 88.33±0.33 50.91±0.45 67.46±0.42</td></tr><tr><td>4.1.2 BASE LEANERS</td><td/><td/><td/><td/></tr><tr><td colspan="5">We used ProtoNets Snell et al. (2017), MetaOptNet-SVM Lee et al. (2019) (we write it as M-SVM)</td></tr><tr><td colspan="5">and Ridge Regression Differentiable Discriminator (R2-D2) Bertinetto et al. (2018) as basic methods</td></tr><tr><td>to verify the effective of Task Aug.</td><td/><td/><td/><td/></tr></table>

Comparison to the average accuracies (%) with 95% confidence intervals between the methods with and without Task Aug on CIFAR-FS 5-way. The results are better than its compared results are highlighted.

Table content: <table><tr><td>Method</td><td>1-shot Baseline Task Aug</td><td>5-shot Baseline Task Aug</td></tr><tr><td>ProtoNets Snell et al. (2017)</td><td colspan="2">71.88±0.52 74.15±0.50 84.14±0.36 85.37±0.35</td></tr><tr><td>ProtoNets (+ens)</td><td colspan="2">73.95±0.51 75.89±0.48 85.72±0.35 87.33±0.33</td></tr><tr><td>ProtoNets (+val)</td><td colspan="2">73.20±0.51 75.10±0.49 85.29±0.35 86.53±0.34</td></tr><tr><td>ProtoNets (+ens+val)</td><td colspan="2">76.05±0.49 77.28±0.47 86.88±0.34 88.24±0.33</td></tr><tr><td>M-SVM Lee et al. (2019)</td><td colspan="2">71.52±0.51 72.95±0.48 84.01±0.36 85.91±0.36</td></tr><tr><td>M-SVM (+ens)</td><td colspan="2">74.12±0.50 75.85±0.47 85.85±0.34 87.73±0.33</td></tr><tr><td>M-SVM (+val)</td><td colspan="2">72.42±0.50 73.13±0.47 84.94±0.36 86.94±0.34</td></tr><tr><td>M-SVM (+ens+val)</td><td colspan="2">75.91±0.48 76.75±0.46 87.15±0.34 88.38±0.33</td></tr><tr><td colspan="3">R2-D2 Bertinetto et al. (2018) 72.27±0.51 74.42±0.48 84.60±0.36 86.02±0.35</td></tr><tr><td>R2-D2 (+ens)</td><td colspan="2">75.06±0.50 76.51±0.47 86.11±0.34 87.63±0.34</td></tr><tr><td>R2-D2 (+val)</td><td colspan="2">73.52±0.50 76.02±0.47 85.39±0.36 86.73±0.34</td></tr><tr><td>R2-D2 (+ens+val)</td><td colspan="2">76.40±0.49 77.66±0.46 87.04±0.34 88.33±0.33</td></tr></table>

Comparison to the average accuracies (%) with 95% confidence intervals between the methods with and without Task Aug on FC100 5-way. The results are better than its compared results are highlighted.Bertinetto et al. (2018) containing all 100 classes from CIFAR-100 Krizhevsky et al. (2010) is proposed as few-shot classification benchmark recently. These classes are randomly divided into training classes, validation classes and test classes. The three types contain 64, 16 and 20 classes, respectively. There are 600 nature color images of size 32 × 32 in each class.

Table content: <table><tr><td>Method</td><td>1-shot Baseline Task Aug</td><td>5-shot Baseline Task Aug</td></tr><tr><td>ProtoNets Snell et al. (2017)</td><td colspan="2">37.53±0.40 39.07±0.40 51.43±0.39 54.49±0.40</td></tr><tr><td>ProtoNets (+ens)</td><td colspan="2">40.04±0.41 42.23±0.42 54.24±0.40 57.47±0.40</td></tr><tr><td>ProtoNets (+val)</td><td colspan="2">43.63±0.43 45.85±0.45 61.16±0.42 61.05±0.41</td></tr><tr><td>ProtoNets (+ens+val)</td><td colspan="2">47.16±0.46 48.05±0.46 63.64±0.43 65.63±0.42</td></tr><tr><td>M-SVM Lee et al. (2019)</td><td colspan="2">40.50±0.39 40.54±0.39 54.83±0.40 56.88±0.41</td></tr><tr><td>M-SVM (+ens)</td><td colspan="2">43.24±0.42 44.00±0.42 58.49±0.41 60.01±0.41</td></tr><tr><td>M-SVM (+val)</td><td colspan="2">46.72±0.45 47.13±0.44 62.99±0.42 63.52±0.41</td></tr><tr><td>M-SVM (+ens+val)</td><td colspan="2">49.50±0.46 49.83±0.45 66.37±0.42 67.14±0.42</td></tr><tr><td colspan="3">R2-D2 Bertinetto et al. (2018) 40.66±0.41 41.34±0.41 55.85±0.39 55.83±0.39</td></tr><tr><td>R2-D2 (+ens)</td><td colspan="2">43.27±0.42 44.45±0.43 58.01±0.40 59.82±0.41</td></tr><tr><td>R2-D2 (+val)</td><td colspan="2">47.12±0.44 47.78±0.44 63.32±0.40 63.73±0.43</td></tr><tr><td>R2-D2 (+ens+val)</td><td colspan="2">49.92±0.45 50.91±0.45 65.58±0.42 67.46±0.42</td></tr><tr><td colspan="2">4.2 EVALUATION ON CIFAR DERIVATIVES</td><td/></tr><tr><td>The CIFAR-FS</td><td/><td/></tr></table>

The average accuracies (%) with 95% confidence intervals on miniImageNet. * Result fromLee et al. (2019). The best results are highlighted. Here only list the best results of previous works due to the shortage of space.

Table content: <table><tr><td>Method</td><td>1-shot</td><td>5-shot</td></tr><tr><td>Gidaris &amp; Komodakis (2018)</td><td colspan="2">56.20±0.86 73.00±0.64</td></tr><tr><td colspan="3">TADAM Oreshkin et al. (2018) 58.50±0.30 76.70±0.30</td></tr><tr><td>LEO Rusu et al. (2018)</td><td colspan="2">61.76±0.08 77.59±0.12</td></tr><tr><td>ProtoNets  *  Snell et al. (2017)</td><td colspan="2">59.25±0.64 75.60±0.48</td></tr><tr><td>M-SVM Lee et al. (2019)</td><td colspan="2">64.09±0.62 80.00±0.45</td></tr><tr><td>M-SVM (+ens+val) (our)</td><td colspan="2">65.38±0.45 82.13±0.31</td></tr><tr><td>R2-D2 (+ens+val) (our)</td><td colspan="2">65.95±0.45 81.96±0.32</td></tr></table>

Comparison to the average accuracies (%) with 95% confidence intervals between the methods with and without Task Aug on miniImageNet 5-way. The results are better than its compared results are highlighted.

Table content: <table><tr><td>Method</td><td>1-shot Baseline Task Aug</td><td>5-shot Baseline Task Aug</td></tr><tr><td>ProtoNets Snell et al. (2017)</td><td colspan="2">58.67±0.48 60.52±0.48 75.24±0.37 77.00±0.36</td></tr><tr><td>ProtoNets (+ens)</td><td colspan="2">62.12±0.48 63.69±0.47 78.11±0.34 79.77±0.34</td></tr><tr><td>ProtoNets (+val)</td><td colspan="2">60.13±0.48 62.22±0.49 76.98±0.36 77.59±0.37</td></tr><tr><td>ProtoNets (+ens+val)</td><td colspan="2">63.84±0.48 65.04±0.48 79.54±0.35 80.60±0.34</td></tr><tr><td>M-SVM Lee et al. (2019)</td><td colspan="2">60.02±0.45 62.12±0.44 77.85±0.34 78.90±0.34</td></tr><tr><td>M-SVM (+ens)</td><td colspan="2">63.44±0.45 64.56±0.44 80.18±0.32 81.35±0.32</td></tr><tr><td>M-SVM (+val)</td><td colspan="2">61.58±0.45 63.14±0.45 78.65±0.34 79.97±0.33</td></tr><tr><td>M-SVM (+ens+val)</td><td colspan="2">64.74±0.45 65.38±0.45 81.39±0.32 82.13±0.31</td></tr><tr><td colspan="3">R2-D2 Bertinetto et al. (2018) 60.57±0.44 62.32±0.45 77.44±0.34 78.81±0.34</td></tr><tr><td>R2-D2 (+ens)</td><td colspan="2">63.72±0.44 64.79±0.45 79.90±0.33 81.08±0.32</td></tr><tr><td>R2-D2 (+val)</td><td colspan="2">62.82±0.45 62.64±0.44 78.61±0.35 79.58±0.33</td></tr><tr><td>R2-D2 (+ens+val)</td><td colspan="2">65.50±0.45 65.95±0.45 81.34±0.32 81.96±0.32</td></tr></table>

Peer review (iclr2020):
The authors propose creating new classes for augmented data instead of the traditional approach of adding them to the original source class. The hope is that a more diverse set of tasks will be created, thus better approximating some unseen task in few-shot learning. They test this idea by rotating images to augment the data and then compare the two approaches. Their experiments were conducted using several data sets and a few different base learners. In nearly all the results, they show that their approach performs better. I find Figure 3 difficult to read because of the separation of the 1-shot and 5-shot classes. (The figure feels a bit clumsy to read.) I*m also unclear what that section contributes to the paper since it is so short. Why only use rotations to augment? I weakly accept this paper because the simple idea seems elegant and appears to improve performance. Since I know little about the field, I will defer to other, more qualified reviewers. I would like to see more experimentation though since this is purely an experimental paper (theory would be wonderful).