Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to learn spatial embedding for coordinates based on H3 hexagons and convolutional autoencoders using the OSM’s tag data. 
strength_and_weaknesses: S1. The problem is important and using convolutional autoencoders in building global spatial embeddings seems to be novel.

W1. The method is very similar to Hex2Vec (both based on hexagons and OSM data). The performance improvement in the experiments is also very marginal.

W2. The scale effect is very important for spatial big data analysis. However, the current method only learns representation for a fixed scale of the spatial region (H3 hexagon), which may not be useful for practical spatial data analysis.

W3. The evaluation is only conducted for US and Italy, which cannot well support the motivation that the learned embedding is for the whole earth.

clarity,_quality,_novelty_and_reproducibility: clarity: the paper's basic idea is clear

novelty: using a hexagon convolution encoder seems to be novel, but existing work has already used the same data for learning spatial embeddings and achieved similar performance (Hex2Vec).

quality: more experiments on multi-scale/global spatial data analysis tasks are preferred.
summary_of_the_review: In brief, learning global spatial embedding is a valuable task. However, the currently proposed method and the used data lack significant contribution beyond existing work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The author proposed to a new geo-encoding method called GeoVeX. The idea is to train an autoencoder on the OSM data projected to H3. Experiments show that GeoVeX outperforms baselines like Hex2Vec and Space2Vec.

strength_and_weaknesses: Strength
- The paper is clearly written and experiments demonstrated GeoVeX's improvement.

Weakness
- The novelty of the paper is limited. The general idea (using OSM + H3) is very similar to the Hex2Vec paper.
- The paper lacks ablation study on understanding why the proposed method work.

clarity,_quality,_novelty_and_reproducibility: 3. Clarity, Quality, Novelty And Reproducibility

Clarity: The paper is clear
Quality: The technical quality is sound
Novelty: The novelty of the paper is very limited given the existence of Hex2Vec and Space2Vec.
Reproducibility: The author can provide more details in how they are training the mdoels in the Appendix.

summary_of_the_review: Vote for weak reject due to lack of novelty.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors proposed GeoVEX as a framework for global representation learning in gespatial settings. The apprach leverages the H3 geospatial indexing system and data from Open Street Maps to create an embedding for each location on earth. An autoencoder like network architecture was introduced for the embedding generation which includes a novel hexagonal convolutional operation and adds a Zero-Inflated Poisson probabilistic output layer. The learnt representations were tested for the tasks of vacation rentals price prediction and temperature interpolation. 
strength_and_weaknesses: -- Strengths:
- Approach is novel, technically sound, and relevant to many applications
- Great potential impact
- Article is well written and set in context

-- Weaknesses:
- Not enough evaluation was done to fully test the power of the learnt representations. Only two downstream tasks were tested without comparing performance to other strong approaches
- Further ablation studies are needed to understand better the proposed approach. How important is the use of the ZIP output layer? What is the performance without it? How GeoVex would perform using other geographic systems?
clarity,_quality,_novelty_and_reproducibility: Please refer to the strengths and weaknesses section. Overall, the paper is well written and novel. 
summary_of_the_review: There is enough contribution for the paper to be accepted but it would be stronger with a better set of downstream task evaluations and comparisons to other approaches for those.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to learn general-purpose location embeddings which are useful for downstream geospatial prediction tasks. The idea is to learn this representation by binning geographic entity occurrence data from OpenStreetMap (OSM) and applying a convolutional autoencoder with a special custom decoder head. The binning and convolution are done on a hexagonal grid. The proposed method (GeoVeX) is compared against two competing approaches to geospatial representation learning (Hex2Vec and Space2Vec) on two datasets. 
strength_and_weaknesses: # Strengths
* Learning geospatial representations that generalize to many downstream tasks is an interesting and worthwhile goal. 
* The idea of autoencoding OSM data is a reasonable approach to geospatial representation learning.
* The proposed ZIP head is novel to the best of my knowledge. 

# Weaknesses
* A key claim of the paper is that it produces "the first geospatial vectors trained on worldwide open data" - this does not seem to be true as written. For instance, [mac2019presence] learns embeddings for each location on earth from freely available data from iNaturalist. See also [sheehan2019predicting], which represents locations using embeddings of nearby geo-located Wikipedia articles. 
* A key claim of the paper is that the proposed method learns "task-agnostic global geospatial vectors." Are they task agnostic? It is not obvious to me that using OSM data makes it "task agnostic" automatically, since OSM contains a limited set of categories with a particular focus on certain land use / land cover categories. More broadly, if the claim is that the proposed method learns significantly more general features than prior methods, it would probably make sense to demonstrate this on more than two datasets. 
* A key claim of that paper is that it is "the first use of hexagonal convolutions within autoencoder architectures." First, I would like to be convinced that this is significant by itself. Can't hexagonal convolutions can be used with any convolutional architecture? Is there a reason we wouldn't expect it to work for autoencoders? Second, it seems like [hahner2022mesh] may have done this first - please comment and clarify. 
* It's a bit tricky to interpret the results in Table 1 and Table 2 without more context. The magnitude of the performance differences between the methods seems small, and the ranking of methods seems to depend on the metric. Given this, it may be important to report average results across a few training runs (both the representation learning and the GBM) to determine whether we are observing robust performance differences or random fluctuations.
* GeoVeX several components, including the use of hexagonal spatial units, the ZIP head, and special terms in the loss function, none of which are characterized with an ablation study. The relative importance of these components is therefore unclear. 

Minor comments:
* The paper claims that a linear probe does not work well in this context because GBM does better than the linear probe when combined with the baseline features. But isn't this what we'd expect if the baseline features are not very good? We would expect linear regression to work better with more powerful features. 
* Why are GeoVeX features always combined with task-specific features (Figure 1)? Why not evaluate the utility of the features on their own for downstream tasks? 
* In the related work section, the paper notes that Space2Vec "uses trigonometric functions with different frequencies to model a given position in space... However, the embeddings are not learnt based on the nearby geographical entities..." - note that [mac2019presence] does both of those things, where the geographic entities are species observations. 

@inproceedings{sheehan2019predicting,
  title={Predicting economic development using geolocated wikipedia articles},
  author={Sheehan, Evan and Meng, Chenlin and Tan, Matthew and Uzkent, Burak and Jean, Neal and Burke, Marshall and Lobell, David and Ermon, Stefano},
  booktitle={Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery \& data mining},
  pages={2698--2706},
  year={2019}
}

@inproceedings{mac2019presence,
  title={Presence-only geographical priors for fine-grained image classification},
  author={Mac Aodha, Oisin and Cole, Elijah and Perona, Pietro},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={9596--9606},
  year={2019}
}

@inproceedings{hahner2022mesh,
  title={Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes},
  author={Hahner, Sara and Garcke, Jochen},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  pages={885--894},
  year={2022}
}
clarity,_quality,_novelty_and_reproducibility: # Clarity
* The paper is generally well-written, with clear tables and figures. 

# Novelty
* The proposed ZIP head for the decoder is novel to the best of my knowledge. 
* Some of the other novelty claims made by the paper are questionable - the details are provided in the "Weaknesses" section above. 

# Reproducibility
* There is not sufficient information in the paper to reproduce the results e.g. hyperparameter settings, tuning procedures. The paper makes reference to an appendix, but it does not seem to be included. Since this paper is meant to demonstrate the superiority of the proposed method, is important to provide details about how all methods (especially competing methods) were trained and tuned. 
* How was the data split for the downstream tasks? Uniformly at random? 
* What is the embedding dimensionality for GeoVeX, Space2Vec, and Hex2Vec? If the dimensions are different, does this provide unfair advantages to some methods? 

summary_of_the_review: This paper tackles an interesting problem and I enjoyed reading it. However, some of the (rather strong) claims made by the paper seem to be inadequately supported or incorrect. In addition, there are some important details missing (hyperparameters, tuning procedures, split construction) which makes it difficult to take the experimental results at face value.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an algorithm for learning geospatial embeddings based on an autoencoder with hexagonal convolutions. The model is trained using a Poisson loss. Empirical benefits are shown over 2 downstream tasks. The reviewers uniformly agree on the importance of the problem being studied. There is also a mixed acknowledgement of the empirical results in the paper and the novelty --- some reviewers appreciated these aspects whereas others find the paper lacking on both accounts. There was a key question regarding the differences with hex2vec which the authors aptly clarified. The major pending concerns with the paper are on (a) the limited empirical evaluations w.r.t downstream tasks and comparisons with related baselines [sheehan2019predicting] and [mac2019presence]; (b) the strong claims on the generality of OSM in comparison with e.g., Wikipedia; (c) the lack of ablations for the different components of the work (simply citing other works is insufficient). I believe all these concerns are important for full justice to the generality claimed in this work, so I'd encourage the authors to conduct further experiments to address them for a future submission.
justification_for_why_not_higher_score: The empirical evaluation is limited as highlighted in the metareview. Given this is a primarily empirical work, there needs to be robust ablations, comparisons, and discussions around various design choices.
Decision: Reject