TASK DEFINITION: In this task, you are given a paper review. Based on the review, your job is to identify language and generate "en" if the review is in English or generate "es" if the review is in Spanish. Note that URLs in the text have been replaced with [Link].
PROBLEM: This paper explores different strategies for instance-level image retrieval with deep CNNs. The approach consists of extracting features from a network pre-trained for image classification (e.g. VGG), and post-process them for image retrieval. In other words, the network is off-the-shelf and solely acts as a feature extractor. The post-processing strategies are borrowed from traditional retrieval pipelines relying on hand-crafted features (e.g. SIFT + Fisher Vectors), denoted by the authors as "traditional wisdom".  Specifically, the authors examine where to extract features in the network (i.e. features are neurons activations of a convolution layer), which type of feature aggregation and normalization performs best, whether resizing images helps, whether combining multiple scales helps, and so on.   While this type of experimental study is reasonable and well motivated, it suffers from a huge problem. Namely it "ignores" 2 major recent works that are in direct contradictions with many claims of the paper ([a] "End-to-end Learning of Deep Visual Representations for Image Retrieval" by  Gordo et al. and [b] "CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples" by Radenović et al., both ECCV'16 papers). These works have shown that training for retrieval can be achieved with a siamese architectures and have demonstrated outstanding performance. As a result, many claims and findings of the paper are either outdated, questionable or just wrong.  Here are some of the misleading claims:     - "Features aggregated from these feature maps have been exploited for image retrieval tasks and achieved state-of-the-art performances in recent years."   Until [a] (not cited), the state-of-the-art was still largely dominated by sparse invariant features based methods (see last Table in [a]).      - "the proposed method [...] outperforms the state-of-the-art methods on four typical datasets"   That is not true, for the same reasons than above, and also because the state-of-the-art is now dominated by [a] and [b].      - "Also in situations where a large numbers of training samples are not available, instance retrieval using unsupervised method is still preferable and may be the only option.".   This is a questionable opinion. The method exposed in "End-to-end Learning of Deep Visual Representations for Image Retrieval" by Gordo et al. outperforms the state-of-the-art on the UKB dataset (3.84 without QE or DBA) whereas it was trained for landmarks retrieval and not objects, i.e. in a different retrieval context. This demonstrates that in spite of insufficient training data, training is still possible and beneficial.    - Finally, most findings are not even new or surprising (e.g. aggregate several regions in a multi-scale manner was already achieved by Tolias at al, etc.). So the interest of the paper is limited overall.  In addition, there are some problems in the experiments. For instance, the tuning experiments are only conducted on the Oxford dataset and using a single network (VGG-19), whereas it is not clear whether these conditions are well representative of all datasets and all networks (it is well known that the Oxford dataset behaves very differently than the Holidays dataset, for instance). In addition, tuning is performed very aggressively, making it look like the authors are tuning on the test set (e.g. see Table 3).   To conclude, the paper is one year too late with respect to recent developments in the state of the art.

SOLUTION: en

PROBLEM: The reviewers seem to agree that the framework presented is not very novel, something I agree with.  The experiments show that the low rank + diagonal parameterization can be useful, however. The paper could be improved by making a more tightened message, and clearer arguments. As it currently stands, however it does not seem ready for publication in ICLR.

SOLUTION: en

PROBLEM: The responses to the pre-review questions are not strong; especially w.r.t. the question about dataset density and why the dataset had to be subsampled, the authors responded that subsampling is common in recommender systems work, including the papers cited. This isn't a particularly strong justification of why subsampling is a good idea, and in particular doesn't answer the question of "how would the results look without subsampling," which I think is a question that could easily have been answered directly.  Especially given that the goal of dealing with the cold-start issue is so heavily emphasized in the paper, in seems odd to sample the data to reduce sparsity.  Other than that, the pre-review questions seem to have been answered satisfactorily.  The contribution of the paper is to propose user and item embedding methods, as a means of learning complex non-linear interactions between users and items. This is fairly similar to recent work on deep RS, though the network formulation has some differences.  Overall this is an reasonably put together paper that makes a contribution in an important area, though there are still some shortcomings that should be addressed, namely: 1) The evaluation is unusual. Recall@M is the only result reported, though this is not usually an evaluation seen in recommender systems research. At the very least other performance measures (rmse or AUC) should be reported for completeness, even if the results are not strong 2) Given that the contribution is fairly simple (i.e., the "standard" recommender systems task, but with a new model) it's a shame that unusual data samples have to be taken. This should be a case where it's possible to report results against competing methods using *exactly* the same data they used, and exactly the same error measure, for the fairest comparison possible.  Without the above it's hard to tell how much the performance improvements are really due to the method being better, versus the choice of datasets and the choice of loss functions.

SOLUTION:
en