Extended Agriculture-Vision: An Extension of a Large Aerial Image Dataset for Agricultural Pattern Analysis
Authors: Jing Wu
David Pichler
Daniel Marley
Naira Hovakimyan
David A Wilson
Jennifer Hobbs
Venue: Accepted by TMLR ()

Abstract: A key challenge for much of the machine learning work on remote sensing and earth observation data is the difficulty in acquiring large amounts of accurately labeled data. This is particularly true for semantic segmentation tasks, which are much less common in the remote sensing domain because of the incredible difficulty in collecting precise, accurate, pixel-level annotations at scale. Recent efforts have addressed these challenges both through the creation of supervised datasets as well as the application of self-supervised methods. We continue these efforts on both fronts. First, we generate and release an improved version of the Agriculture-Vision dataset  (Chiu et al., 2020b) to include raw, full-field imagery for greater experimental flexibility. Second, we extend this dataset with the release of 3600 large, high-resolution (10cm/pixel), full-field, red-green-blue and near-infrared images for pre-training. Third, we incorporate the Pixel-to-Propagation Module Xie et al. (2021b) originally built on the SimCLR framework into the framework of MoCo-V2 Chen et al.(2020b). Finally, we demonstrate the usefulness of this data by benchmarking different contrastive learning approaches on both downstream classification and semantic segmentation tasks. We explore both CNN and Swin Transformer Liu et al. (2021a) architectures within different frameworks based on MoCo-V2. Together, these approaches enable us to better detect key agricultural patterns of interest across a field from aerial imagery so that farmers may be alerted to problematic areas in a timely fashion to inform their management decisions. Furthermore, the release of these datasets will support numerous avenues of research for computer vision in remote sensing for agriculture.

Published in Transactions on Machine Learning Research (03/2023)
Extended Prricllture-Vision: An Extension of a Large Aerial Image Dataset for Agricultural Pattern Analysis
Jing Wu University of Nliniis Urbena- Channnng
jangnuddrlinooi.edl
David Pichler Intelinair
danid.piicherriinnccaaaa..ccm
Daniel Marley
daniel@intelinaiir.om
Intelinair
David Wilson
davidd@inelinaii.com
Intelinair
Naira Hovakimyan
mhovakim edu
University of Ilinois Urbona- Champoign
Intelinair
Jennifer Hobbs
@ganlfrrhobbboD&@@mami... com
Intelinair
Reviewed on OpenReview:
https: // openreview. net/ forum? id= v5juDLaf00
Abstract
key challenge for much of the machine learning work on remote sensing and carth ob- servation data is the difficulty in acquiring large amounts of accurately labeled data. This is particularly true for semantic segmentation tasks, which are much less common in the remote sensing domain because of the incredible difficulty in collecting precise, accurate, pixel-level annotations at scale. Recent efforts have addressed these challenges both through the creation of supervised datasets as well as the application of self-supervised methods. We continue these efforts on both fronts. First, we generate and release an improved version of the Agriculture- Vision dataset (Chiu al. 2020b) to include raw, full-field imagery for greater experimental flexibility. Second, extend this dataset with the release of 3600 large, high-resolution (10em/pixcl) full-field, red-green-blue and near-infrared images for pre-training. Third, incorporate the Pixel-to-Proppe Propagation Module (Xie et. al.] 2021b) originally built on the SimCLR framework into the framework of MoCo-V2 (Chen et al, 2020b). Finally, we demonstrate the uscfulncss of this data by benchmarking different con- trastive learning approaches on both downstream classifications and semantic segmentation lasks. We explore both CNN and Swin Transformer (T.iu al. 2021a) architectures within different frameworks based on MoCo-V2. Together, these approaches enable us to better detect key agricultural patterns of interest across field from acrial imagery that farmers may be alerted to problematic areas in timely fashion to inform their management deci- sions. Furthermore, the relcase of these datasets will support numerous avenues of rescarch for computer vision in remote sensing for agriculture.
1 Introduction
Massive annotated datasets like ImageNet have fostered the development of powerful and robust deep- learning models for natural images (Deng al. 2009: He et. al. 2016 Simonyan & Zisserman 2014

---

Published in Transactions on Machine Learning Research (03/2023)
Krizhevsky et al. lime-consuming, and may be infeasible in some domains or for certain lasks. Simultaneously, vast amounts of unlabeled data exist in most domains. Among different self-supervised learning methods(Chaane
2012
Russakovsky al. 2015): However, creating large complex datasets is costly,
et al. Albelwi 2022 Singh ct al.) 2018 Ziegler & Asano 2022 Contrastive learning has recently emerged as an encouraging candidate for solving the need for large labeled He al.
2022 Khan et al.) 2022 Ziegler & Asano 2022
datasets (Grill et al. 2020 Caron al.
2020
2018
2020). Through pre-training, these approaches
open up the possibility of using unlabeled images as its own supervision and transferring in-domain images
to further downstream tasks (Tian et al.
2020 Ayush et al.
2020
While natural scene imagery largely dominates the rescarch landscape in terms of vision algorithms, datasets and benchmarks, the rapid increase in quantity and quality of remote sensing imagery has led to significant
advances in this domain as well (Kelcey & Lucieer 2012 Maggiori et al.) 2017 Ramanath et al.) 2019
Xia et al. 2017). Coupled with deep neural networks, remote sensing has achieved exceptional success in multiple domains such as natural hazards assessment (Van Westen 2013). climate tracking (Rolnick)
et al. 2019 Yang et al. 2013); and precision agriculture (Mulla 2013 Seelan et al. 2003 Wu et. al. 2022;
Barrientos et al. 2011 Gitelson et al. 2002). However, obtaining large quantities of accurate annotations is especially challenging for remote sensing tasks, particularly for agriculture, as objects of interest tend to be very small, high in number (perhaps thousands per image), possess complex organic boundaries, and may require channels beyond red-green-blue (RGB) to identify.
Approaches developed initially for natural images may work well on remote sensing imagery with only minimal modification. However, this is not guaranteed due the large domain gap. Additionally, initial methods may fail to exploit the unique structure of earth observation data, such geographic consistency seasonality (Mañas et al. 202Ip. Explicitly benchmarking approaches on domain-relevant data is critical. In this work, we focus primarily on the Agriculture- Vision (AV) dataset (Chiu et al., 2020b): a large, multi- spectral, high-resolution (10 cm/ /pixeelx, labeled remote sensing dataset for semantic segmentation. Unlike low-resolution public satellite data, this imagery enables within-ficld identification of key agronomic patterns such as weeds and nutrient deficiency. While this dataset is noted for its size, most aerial agriculture datasets are quite small. Therefore we leverage the large amounts of un-annotated data which is readily available in this domain, benchmark several self-supervised approaches whose inductive bias reflects the structure of this data, and evaluate the impact of these approaches in more data-limited settings.
Together, our contributions are as follows:
We relcase a full-field version of the Agriculturo-Vissin dataset to further encourage broad agricul- lural research in patlern analysis.
We release over terabytes of unlabeled, full-field images from more than 3600 full-field images enable unsupervised pre-training.
We benchmark self-supervised pre-training methods based on momentum contrastive learning and evaluate their performance on downstream classification and semantic segmentation tasks with vari- able amounts of annotated data.
We perform benchmarks using both CNN and Swin Transformer backbones.
We incorporate the Propaagtton Module (Xie et al. 2021bb (PPM), originally built on framework and evaluate its
SimCLR (Chen et al. 2020a)). into the MoCo-V2 (Chen al. performance.
2020b
We adapt the approach of Seasonal Contrast (SeCo) from Mañas et al. 2021 for this dataset, which contains imagery only during the growing season to address the spatiotemporal nature of the raw data specifically.
2 Related Work
2.1
Constrastive Learning
Unsupervised and self-supervised learning (SSL) mothods have proven to be very successful for pre-training
deep neural networks (Erhan et al. 2010 Bengio
2012 Mikolov al.) 2013 Devlin et al. 2018). Recently,
methods like MoCo (He et. al.] 2020 Chen al.] 2020b). SimCI.R (Chen et al. 2020a), BYOL Grrll et al.
2

---

Published in Transactions on Machine Learning Research (03/2023)
2020) and others such Bachman et al. 2019 Henaff 2020 Li et al. (020) based on contrastive learning methods have achieved state-of-the-art performance. These approaches seek to learn by encouraging the attraction of different views of the same image ("positive pairs") as distinguished from "negative pairs" from different images (Hadsell et al.) 2006). Several approaches have sought to build on these base frameworks by making modifications that better incorporate the invariant properties and structure of the input data or task output. Specifically pertinent to the current work, Xie et al. (2021b) extended the SimCLR framework through the incorporation of a performance on downstream tasks requiring dense pixel predictions. Mañas et al. (2021) combined multiple encoders to capture the time and position invariance in downstream remote sensing tasks.
module and additional pixel-level losses to improve
2.2
Remote Sensing Datasets
Aerial images have been widely explored over the past few decades (Cordts et al., 2016 Everingham et al, Gupta et al., 2019 Lin et al. 2014 Zhou et al., 2017). but the datasets for image segmentation typically focus on routine, ordinary objects or street scenes (Deng et al.) 2009). Many prominent datasets including Inria Aerial Image (Maggiori et al. 2017), EuroSAT (Helber al. 2019). and DeepGlobe Building (Demir 2018 are built on low-resolution satellite (e.g. Sentinel-1, Sentinel-2, MODIS, Landsat) and only have limited resolutions that vary from 800 cm/ /pixel to 30 cm/pixel and can scale up to 5000x5000 pixels. Those
2010
et al.
datasets featuring segmentation tend to explore land-cover classification or change detection DDudtt et al.
2018 Sumbul et al. 2019).
Pertaining aerial agricultural imagery, datasets Lend to be either low-resolution (>10 m/pixel) satel- lite (Tseng et al. 2021 Feng & Bail 2019) or very high-resolution (<1 cm cm/pixel) imagery taken from UAV
on-board farming equipment (Haug & Ostermann 2014 Olsen et al. 2019). The Agriculture-Vision
dataset (Chiu et al.
2020b a) introduced large, high-resolution (10 cm pixel) dataset for segmentation,
bridging these two alternate paradigms.
3 Datasets
3.1
Review and Reprocessing of Agriculture- Vision Dataset
The original AV dataset (Chiu et al. 2020b) consists of 94,986 labeled high-resolution (10-20 em/pixel) RGB and near-infrared (NIR) aerial imagery of farmland. Special cameras were mounted to fixed-wing aircraft and flown over the Midwestern United States during the 2017-2019 growing seasons, capturing predominantly corn and soybean fields. Each field was annotated for nine patterns described in the supplemental material. After annotation, 512 512 tiles were extracted from the full-field images and then pre-processed and scaled, While this pro-psscss produces uniformly curated dataset, naturally discards important information about the original data.
To overcome this limitation, we obtained the original raw, full-field imagery. We are releasing this raw data full-field images without any tiling, as it has been demonstrated to be beneficial to model performance (Chiu et al. 2020a). A sample image is shown in Figure (left). The original datasct can be recreated from this new datasel by extracting the tiles at the appropriate pixel coordinates provided in the data manifest.
3.2 Raw Data for Pre-training
We identified 1200 fields from the 2019-2020 growing seasons collected in the same manner as in Section 3.1 For each field, selected three images, referred to flights, taken at different times in the growing season, resulting in 3600 raw images available for pre-training. We elect to include data from 2020 even though it is not a part of the original supervised dataset because it is of high quality, similar in distribution to 2019, and we wish to encourage exploration around incorporating different source domains into modeling approaches as this is a very central problem to remote sensing data. We denote this raw imagery plus the original supervised dataset (in full-field format) as the "Extended Agriculture- Vision Dataset" (AV+); it will be made publicly available. The statistics of AV+ compared with AV are demonstrated in Table 1
3

---

Published in Transactions on Machine Learning Research (03/2023)
4: Ignore
3: Crop
Weed
Unmanaged
Figure Left: Full-field imagery (RGB-only) constructed from the AV dataset. field of this size is approximately 15,000x15,000 pixels which can yield many smaller tiles. Right: Sample imagery and labels for the fine-grained segmentation task.
One characteristie of remole sensing is data revisiting: capturing images from the same locations multiple times. Through data revisiting, the temporal information serve additional dimension of variation beyond the spatial information alone. In AV+, typical revisit time ranges from seven days to six months, capturing a field at different points during the pre-planting, growing, and harvest seasons. We provide an example of a revisit in Figure 3
Table 1: Statistics of Agriculture Vision SasionCCinn Chiu et al. 2020b) and Extended Agriculture Vision(the part of raw imagery). We provide information about the number of images, image size, pixel numbers, color channels and the ground sample resolution (GSD). "cls.", "seg." and "SSL" stand for classification, segmentation and self-supervised learning respectively.
Dataset
# of Images Tasks
Image Size
Channels
Resolution (GSD)
# of pixels
AV
94,986 3600
Cls./Seg. SSL
512 x 512
RGB, NIR RGB, NIR
10/15/20 cm/px
22.6B
AV+
15,000 x 15,000
10/15/20 cm/px
810.0B
3.3
Fine-Grained Segmentation Dataset
Fine-grained segmentation tasks for high-resolution remote sensing data, particularly for agriculture, are often overlooked because of the difficulty in collecting sufficient amounts of annotated data(Monteiro & von Wangenheim 2019: Haug & Ostermann 2015). To explore the transferability of the AV- dataset and SSL methods to a very challenging, lita-limited, in-domain (i.e, same sensor and geography) task, we construct a densely annotated dataset. We collected flights from the 2020 growing season that were not included AV+ for this task. From these flights, 184 tiles with shape 1500x1500 were sclected and densely annotated with four classes: soil, weeds, crops, and un-managed area (e-g. roads, trees, waterways, buildings); an "ignore" label was used to exclude pixels which may unidentifiable due image collection issues, shadows, or clouds. The annotations in this dataset are much more fine- gre-ggainnd than those in the AV+ dataset. For example, whereas the AV+ dataset identifies regions of high weed density as a "weed cluster", this dataset identifies each weed individually at the pixel level and also labels any crop or soil in those regions by their appropriate class. A sample image and annotation are shown Figure I (right).
The fine- ane-geai nature and small dataset size make this a very challenging segmentation task: very young crops often look like weeds, weeds growing among mature crops only detectable through interruption in the larger structure of the crop row, unmanaged areas often contain grasses and other biomass which closely resemble weeds but is not of concern to the grower, and classes are highly imbalanced.
4 Methodology for Benchmarks
In this section, we present multiple methods for pre-training a transfcrable representation on the AV+ dataset. These methods include MoCo-V2 (Chen et al.] (2200)) MoCo-V2 with Pixel-to- Propaggatinn Module (PPM) (Xie et al. 2021b), the multi-head Temporal Contrast based SeCo (Mañas et al. 2021 and
4

---

Published in Transactions on Machine Learning Research (03/2023)
combined Temporal Contrast model with PPM. We also explore different backbones based on ResNet He
et al. 2016 and the Swin Transformer architecture [Lu et al. 2021a).
4.1 Momentum Contrast
MoCo-V2 is employed the baseline module for the pre-training task. Unlike previous work focusing only on RGB channels (He et al. 2020 Mañas et al. 2021 Chen et al. 2020b), we include the information and learn representations from RGB and NIR channels. In each training step of MoCo, a given training example x is augmented into two separate views, query a and key x². An online network and a momentum-updatted offline network, map these two views into close embedding spaces q = (29) and k+ = fx(²²) accordingly; the query q should be far from the negative keys k- coming from a random subset of data samples different from t. Therefore, MoCo can be formulated as a form of dictionary lookup in which k+ and k- are the positive and negative keys. We define the instance-level loss Linst with temperature parameter for scaling (Wu et al. 2018) and optimize the dictionary lookup with InfoNCE (Oord et al. 2018):
cxplg- k-
Linst - log Σx- expla k- /-/T///////>>> + exple kt k+/T)
(1)
4.2 Momentum Contrast with Pixel-to- toooap-atoooaiionnn Module
Compared with classical datasets such as ImageNet. (Deng ct al. 2009 COCO (Lin et al. 2014), and LVIS (Gupta al. 2019) in the machine learning community, low-level semantic information from AV+ is more abundant, with regions of interest corresponding more closely to "patterns" (i.e. areas of weed clusters, nutrient deficiency, storm damage) and less to individual instances. Therefore, pre-training MoCo-V V2 beyond image-level contrast should be beneficial to downstream pattern analysis tasks.
4.2.1 Exeeltto-Propggation Module
Xie et al. 22021b) added Pistl-Prnpagation-oohll (PPM) to the SimCLR framework and achieved out- standing results dense downstream tasks. In PPM, the feature of pixel is smoothed to of by feature propagation with all pixels z; within the same image I following the equation:
9
Σ S(x,, G(æ;), ZyÊl
(2)
where G(.) is transformation function instantiated by linear and ReLU layers. S(.,)) .) is similarity function defined as
S(z,, =
0)),
(3)
with a hyper-parameter yy control the sharpness of the function.
4.2.2 Extend Sheeltto-Propagatie Module to MoCo
Notably, previous work (Xie et al. 2021b based on SimCLR requires a large batch size, which is not always achievable. To generalize the PPM and make the overall pre-training model efficient, we incorporate the pixel-level pretext tasks into basic MoCo-V2 models learn dense feature representations. As demonstrated in Figure 2 we add two extra projectors for pixel-level pretask compared with MoCo-V2 The features from the backbones are kept as feature maps instead of vectors to ensure pixel-level contrast. To decide positive pairs of pixels for contrast, cach feature map is first warped to the original image space. Then the distances between the pixel and pixel j from each of the two feature maps are computed and normalized. Given a Soper-parameter 7 (set as 0.7 by default), and j are recognized as one positive pair if their distance is less than т. Then, we can compute the similarity between two pixel-level feature vectors, i.c., smoothed of from PPM and ky from the feature map for each positive pair of pixels and j. Since two augmentation views both pass the two encoders, we loss in symmetric form following Xie al. 2021b
5

---

Published in Transactions on Machine Learning Research (03/2023)
A
B
__orop
X
backbone
PPM
x
x
backbone
P_P1
PPM
X
momentum
backbone
X
InfoNEC
cosine similarity
x
P2
x
cosine similarity
T_crop
X-1
momentum backbone
X
momentum
backbone
P_22
x
P3
x
X
momentum
P2
backbone
P: instance-level projector
P P: pixel-level projector
positive pairs
share weights
PPM: pixel-to-propagation module
Figure 2: A. Diagram of MoCo-V2 with Pixel-to-Propagation Module (MoCo-PixPro). includes a nor- mally updated projector and nomentum-updattee projector. For pixel-level pre-task, P P_P1 P1 is updated by gradient descent and P P2 is the momentum projector. B. Diagram of Temppral Contrast with Pixel-to- Propagation Module (TemCo-PixPPro). Query view 2 and key view εÃ contain both artificial and temporal variance. Query view I" and key view _²1 contain only temporal variance. Query view x and key view =xx only contain artificial variance. Identical cropping Terep is applied to z and r²s, Pixel-level contrast in only computed on x and x²²2 For these two sub-plots, modules in navy blue serve as encoders for feature extraction. Modules in brown are designed for pixel contrast, which includes projectors, pixel propagation modules and the loss being used. Pink modules represent instance-level contrast with embeddings space invariant to all kinds of augmentations. Similarly, modules in green and sky blue mean instance-level contrast but extract features invariant to artificial augmentation and temporal augmentation, respectively.
CpiPPro cos(sa ky) cos(af..k))
During the training, the loss PrixPro from the PPM is integrated with the instance-level loss as shown in the equation 5 These two complementary losses are balanced by a factor a, set to 0.4 in all the experiments (see Supplemental: Additional Results Balance Factor).
L aLinst PiiPPro
(5)
4.3 Temporal Contrast
While a pixel-level pretext task learns representations useful for spatial inference, we would like to learn a representation that takes advantage of the temporal information structure of AV+. Following the work of SeCo (Mañas et al.) 2021), additional embedding sub-spaces that are invariant to time are created. Since the backbones learn temporal-aware features through extra sub-spaces, offers a more precise and general pattern analysis downstream tasks. More specifically, we define a positive temporal pair by obtaining one pair of images from the same area (i.e. of the same field) but at different times as shown Figure 3 We explore whether the structure provided by the temporal alignment of positive temporal pairs provides more semantically significant content than naive artificial transformations (i.c. flipping, shifting) for pre-training.
Unlike in SeCo where images were separated with a constant time (3 months), the time difference bet ween images from our data varies from week to months. We adapt SeCo as follows. First, we randomly select three tiles with 512 512 from the same ficld at identical locations but different times, which will be defined as xt1 t²2 and Ita Only random cropping Terop is applied to the query image generate the query view, i.e, 2º (x")). The first. key view that contains both temporal and artificial variance is defined as ao

---

Published in Transactions on Machine Learning Research (03/2023)
Captured Time: April 2020
Captured Time: October 2020
Figure 3: Visualization of temporal contrast in AV+.
= =T(x"", where the T the typical data augmentation pipeline used in MoCo. The second key contains only temporal augmentation compared with the query view. Therefore, we apply the exact same cropping window applied to the query image, æt (x=o).("2). The third key contains only artificial augmentations, ====T(xfo). Following the MoCo and SeCo learning strategy (He et al. 2020 Mañas et al. 2021 these views can be mapped into three sub-spaces that are invariant to temporal augmentation, artificial augmentation and both variance. In this way, we fully explore the multi-time scale information in AV+ to improve the lemporal sensitivity of encoders further. Since the temporal contrast does not necessarily cross seasons or enforce aligument of seasonality within sub-space, we denote our approach as Temporal Contrast (TemCo).
4.4 Temporal Contrast with Pixel-to- Proppagttooo Module
We create an integrated model (TemCo-PixPro) to capture the dense, spatiotemporal structure of AV+. Concretely, we merge PPM and TemCo into single model to increase the encoders' spatial and temporal sensitivity.
To ensure efficient computation, we do not compute pixel-wise contrastive updates in each temporal sub- space. Instead, we assign two extra projectors for pixel-level contrastive learning. We include the PPM after the online backbone and one of the pixel-level projectors smooth learned features. Then, calculate the similarity of the smooth feature vectors and the momentum encoder features through dot product. We illustrate the overall architecture of this model in Figure 2
4.5 Swin Transformer- Momentum Contrast
While the Swin Transformer achieves superior performance on various computer vision tasks (Liu et al. 2021a b), only very recent work has focused on self-supervised training for vision transformers (ViT) et al. 2021a Li et al., 2021). To the best of our knowledge, study has investigated Swin Trans- former's performance on remote sensing datasets using self-supervised methods. Therefore, we explore a Swin Transformer-basse MoCo for pre-training of AV+ Specifically, we adopt the tiny version of the Swin Transformer (Swin-T) as the default. backbone.
Following most transformer-bast learning tasks, we adopt AdamW Kingma & Ba 2014) for training. Addi- tionally, we incorporate the multiple-head projectors from TemCo and PPM to capture temporal knowledge and pixel-level pretext tasks.

---

Published in Transactions on Machine Learning Research (03/2023)
5 Experiments and Results
We benchmark the performance and transfcrability of the learned representations on four downstream tasks: agricultural pattern classifications, semantic segmentation on AV+, fine-graines semantic segmentation and land-cover classification.
We report the implementation details and results from four models covering two kinds of backbones: ba- sic Momentum Contrast model (MoCo-V2), basic Momentum Contrast with Pixel-to-Prone Preelao--roo Mod- ule (MoCo-PixPro), Temporal Contrast (TemCo), Temporal Contrast with Pixel-to-roopen Prod-to-Propaaation Mod- ule (TemCo-PixPrro) Swin Transformer-Baser MoCo-V2, Swin Hansffrmer-Basser TemCo, and Swin Transformer-Bast Based TemCo-PixPro.
5.1 Pre-training Settings
DDataset. We take AV+ as the dataset to pre-train the backbonos. To be specific, all the raw images are cropped tiles with shape 512 x512 without overlapping, following the settings in Chiu et al. (2020b). Therefore, we eventually have 3 million cropped images. For each tile, it will be fed to the contrastive learning pipelines.
Implementation Details. We train each model for 200 epochs with batch size 512. For ResNet- models, use SGD as the optimizer with weight decay of 0.0001 and momentum of 0.9. The learning rate is set to 0.03 initially and is divided by 10 at epochs 120 and 160. Swin-T models use the AdamW optimizer, following previous work (Xie et al.) 2021a Liu et al. 2021a). The imitial learning rate is 0.001, and the weight decay is 0.05. All the artificial data augmental ions used in this paper follow the work of MoCo-V2 as this data augmentation pipeline archives the optimal performance. These data augmentations include random color jitter, gray-scale transform, Gaussian blur, horizontal flipping, resizing, and cropping.
We use all four channels, RGB and NIR, fully extract the features contained in the dataset. When testing ImageNet-iniitalizze backbones for comparison, we copy the weights corresponding to the Red channel of the pre-trained weights from ImageNet to the NIR channel for all the downstream tasks following the method of Chiu et al. (202bb).
5.2
Downstream Classifications
5.2.1
Preliminaries
The downstream classification task considers nine patterns: Nutrient Deficiency, Storm Damage, Drydown, Endrow, Double plant and Weeds. More details descriptions of these patterns be found in the supplementary. We benchmark and verify the performance of our basic model (MoCo-V2) and its variants on the classification task of the labeled portion of AV+. following three protocols: (i) lincar probing (He
Protocols.
et al.) 2020: Chen et al.) 2020b a Tian et al.) 2020), (ii) non-linear probing (Han et al. 2020). and
finettunig the entire network for the downstream task.
Datasct. For this classification task, there are total of 94,986 tiles with a size of 512 from AV. With a 6/2/2 train/val/test ratio, AV contains 56,944/18,334/11 14,333//99,70 train/val) tein/val/tei images following the exact settings from Chiu et al. (020b)) We train the classifiers the training set and evaluate their performance in the validation set.
5.2.2 Linear Probing
Following standard protocol, we freeze the pre-trained backbone network and train only a lincar head for the downstream task. We train the models for 50 epochs using Adam optimizer with an initial learning rate of 0.0001 and report the top-1 classification validation set.
Figure Asoows the impact of different weight initialization and percentages of labeled data the downstream task. Consistent with previous research (Mañas al.) (002)), there is gap between remote sensing and natural image domains: ImageNet weights are not always optimal choice in this domain. MoCo-PixPro

---

Published in Transactions on Machine Learning Research (03/2023)
ResNet-18
Res Net-50
Swin-T
80
80
80
75-
75
70
70
55
60
55
Peccentage
Percentage
Percentage
ImageNet
MoCo
TemCo
IemCoP
MoCof (Best)
Figure 4: Accuracy under the linear probing protocol on AV+ classification. Results are shown from different pro-training approaches with different backbones.
ResNet-18
ResNet-50
Swin-T
70-
75
75
Acmuuny
70
70
60
0
10
20
30
40
50
80
70
20
90
100
60
10
20
30
50
60
70
80
90
100
60
10
20
60 Parointage labaled (%6
30
50
70
90
100
Percentage of lubaied data (96)
Parcmntage of labaded data
ImageNet
MoCo
TemCo
TemCoP
MoCoP (Best)
Figure 5: Accuracy under the non-linear probing protocol on AV+ classification. Results are shown from different pre-training approaches with different backbones, ResNet-18 (left), ResNet-50 (middle), and Swin-T (right), under different percentages of labeled data for the downstream task.
obtains the highest accuracy for the ResNet-18 backbone. As compare the results of ResNet50 and Swin-T with fully labeled data, all Swin-T models underperformed their CNN counterparts.
5.2.3 Non-Linear Probing
We evaluate the frozen representations with non-linear probing: multi-layer perceptron (MLP) head is trained the classifier for 100 epochs with Adam optimization.
Classification results on AV+ classification under non-linear probing are shown in Figure 5 Consistent with results in the natural image domain (Han et al. 2020), non-linear probing results surpass linear probing. Our SSL weights exceed ImageNet's 's weights by over 5% regardless of the amount of downstream data or backbone type. From the results of ResNet-18, the optimal accuracy between different pre-training strategies comes from either MoCo-PixPre or TemCo-PixPro, which different from linear probing. Overall, MoCo-PixPro performs better than the basic MoCo model across different backbones.
5.2.4 Fine-Tuning
Tuning
Finally, we examine end-to-end tne-tuning with different percentages of labeled AV+ data for classification. We use the same architecture, learning schedule and optimizer as non-lincar probing.
Our SSL weights show outstanding results in the low-data regions (<10% of data). For Res.Net-18, MoCo- PixPro is better than the other models in all cases, whereas other SSL models demonstrate similar per- formance to ImageNet when labeled data is abundant. As we increase the backbone size to ResNet-50, our MoCo and MoCo-PixPro stably outperform ImageNet's 's model across all amounts of data, suggesting a greater capacity to learn domain-relevant features.
In Figure (right), all models perform agrocably well the Swin-T framework compared with weights from ImageNet. While fine-tuning was performed in the same manner the ResNet models for fair comparisons, Swin-T shows the most promising performance in this end-to-end setting.
9

---

Published in Transactions on Machine Learning Research (03/2023)
ResNet-18
Res Net-50
Swin-T
85
85
85
25-
-75
75
63-
63-
65-
60
100
60
100
60
100
Peccentage
Percentage
Percentage
ImageNet
MoCo
TemCo
TemCoP
MoCof (Best)
Figure 6: The accuracy under the end-to-end classification protocol on AV+. Results cover different pre- training approaches and backbones, varying from ResNet-18 (left), ResNet-50 (middle), and Swin-T (right). We also report the model's performance tuned with different percentages of the fully labeled dataset, ranging from one percent to a hundred percent.
5.3
Semantic Segmentation on Extended Agriculture Vision
5.3.1
Preliminaries
Protocols. We continue our benchmarking study by examining their impact on the semantic segmentation approach on AV+ as originally formulated in Chiu et al. (2020b). Again, we apply two protocols for evaluating the learned representations: maintaining a fixed encoder or fine-tuning the entire network.
To naively assess the pre-trained representations, we adopt the simple yet effective U-Net (Ronneberger et al. 2015) framework. Unlike previous work on AV (Chiu el al. (020b), we report results over all the patterns in AV+, including storm damage, ensure an integrated and comprehensive analysis.
First, we evaluate the representation by holding the pre-trained encoder fixed and fine-tuning only the decoder during the supervised learning phase. Similarly, we evaluate the pre-training impact segmentation lasks allowing for fine-tuning of both the encoder and decoder during the supervised learning phase. We train the modols using Adam optimization with an initial lcarning rate of 0.01. The onc-cyele policy {Smith 2017) is used to update the learning rale as in Chiu et al. (2020b). ResNet-11 models are trained for 30,000 sleps, while the larger ResNet-50 and Swin-T models are trained for 120,000 steps allow for sufficient training.
Dataset. For this segmentation task, there are a total of 94,986 tiles with size of 512 512. With the same 6/2/2 train train/val/tern /val/tsst ratio, AV contains 16,944418,334/19, 19,708 train/ main/val/test images following settings from Chiu et al. (2020b) and the previous classification task. We train the classifiers on the training set and evaluate their performance in the validation set.
5.3.2 Segmentation Results
As results are shown in Table 2 MoCo-PixPro performs the best for the ResNet-18 backbone when the encoder remains fixed during supervised training; this result is similar to that seen for classification. This result supports our hypothesis that AV+ has abundant low-level semantic information and including pixel- level pre-task critical for downstream learning tasks. When the encoder is unfrozen during supervised training, the basic MoCo-V2 shows the best results, but not significantly better than TemCo or TemCo- PixPro. By scaling from ResNet-18 18 to ResNet-50, MoCo-PixPro outperforms ImageNet, especially when the encoder remains fixed. Importantly, unlike the bessedd models, the Swin Transformrr-basee MoCo- PixPro shows the best results across all variations in the setting. Another important observation is that the PPM benefits more as we scale up the models from ResNet-18 to ResNet-50 and then Swin-T. As the training epochs are all the same for all the pre-training, smaller backbones like ResNet-18 are more likely to get overfitted. When trained with ResNet-50, the performance drop of MoCo-PixPro very small compared with MoCo. As we move Swin-T, TT, MoCo-PixPro eventually shows the best performance over other methods.
10

---

Published in Transactions on Machine Learning Research (03/2023)
Table 2: Results of Downstream Segmentation Task on AV+ using mean-loU metric
mloU (%)
mloU (%)
mloU (%)
mIoU (%)
Pretrained Weights
Backbone
Fixed
Fixed
Fine-Tume
ine-Tuned
1%
100%
1%
100%
Random
ResNet-18
18.89
21.37
19.02
26.94
ImageNet
ResNet-13 18
19.02
23.39
19.73
29.23
MoCo-V V2
ResNet-18
22.36
27.83
22.53
31.80
R8sNet-18
23.71
30.60
20.04
30.56
TemCo
ResNet-18
23.71
26.85
21.09
31.76
TemCo-PixPro
ResNet-18
22.97
28.60
21.32
31.66
Random
ResNet-50
19.42
21.82
18.71
26.37
ImageNet
ResNet-50
21.21
25.94
20.31
30.52
MoCo-V V2
ResNet-50
24.25
31.03
21.47
31.87
MoCo-PixPro
ResNel-50
25.76
32.35
21.36
31.58
Random
Swin-T
15.89
20.10
22.68
37.14
ImageNet
Swin-T
20.00
22.40
30.96
43.01
MoCo-V V2
Swin-T
25.51
30.60
28.12
41.02
MoCo-PixPro
Swin-T
27.61
32.96
32.06
43.33
5.3.3
Comparison with Agriculture-Viste Results
The AV dataset was benchmarked on a downstream segmentation task with architectures based on the DeepLabV3 (Chen et al. 2018) framework. Since the previous results report mean Intersccion-over-Uiiie (mloU) for agricultural patterns, we re-trained our models using a U-Net architecture (Ronneberger et al. 2015) including on more pattern, i.e., the storm damage. With lightweight U-Net, smaller backbone, and much less training, SwinT-based model outperforms the best results from Chiu et al. (202bb) in the demonstrate the effectiveness of
Table demonstrating the effectiveness of approach. Additionally, pre-training and fine-tuning this mulli-spectral data. AV and AV+ are beyond most conventional images, consisting of NIR-Red- Greee-rreennuui (NRGB) channels. Therefore, we investigate the differences in semantic segmentation performance from multi-spectral images, including regular RGB and RGBN images. Accord- ing to Table NIR channels benefit the segmentation results over different backbones and segmentation methods.
For this eight-class segmentation task, train the ninc-class models using an Adam optimization with initial learning rate of 0.01 and the one-cycle policy (Smith 2017) for the learning rate adjustment. For fair comparisons and to be consistent with the supervised learning settings in Chiu et al. (2020b), we use batch size of 40 and 25,000 iterations with warmup training for 1,000 iterations.
Table 3: Comparison of mloUs between the Prriculture-Vision model and our proposed U-Net-based model Agriculture-Vist validation set.
Methods
Pre-trained Weights
Backbone
Channels
mIOU(%)
# Parameters
FPNíChin et al. 2020b U-Nel
ImageNet
ResNet-101
RGB
40.48
45.10M
MoCo-V2
Swin-T
RGB
44.77
32.40M
U-Net
MoCo-PixPro
Swin-T
RGB
45.92
32.40M
FPN Chiu et al.
2020b
ImageNet
ResNet-100 101
RGBN
43.40
45.
U-Net
MoCo-V2
Swin-T
RGEN
46.15
32.40M
U-Net
MoCo-PixPro
Swin-T T
RGBN
48.75
32.40M
11

---

Published in Transactions on Machine Learning Research (03/2023)
IoU for each model in the fine-graines semantic segmentation task considering different encoder weight initialization, architectures, and weight fixing schemes.
Table 4:
Weights
Architecture
ToU (einh-)
ToU (Fine-Tuned)
Random
ResNet-18
39.05
42.19
ImageNet
ResNet-18
40.81
45.47
MoCo-V2
ResNet-18
44.05
43.97
MoCo-PixPPro
ResNet-18
42.03
44.62
TemCo
ResNet-18
42.30
44.48
TemCo-PixPri
ResNet-18
43.45
43.91
V2
ResNet-50
40.03
40.54
MoCo-V2
Swin-T
40.25
40.00
MoCo-PixPro
Swin-T
37.56
40.67
TemCo.
Swin-T T
39.52
40.26
Ignore
2: Weed
Unmanaged
0: Soll
RGB
ImageNet
MoCo-v2
Labels
Figure A sample output on the fine- ine-grained segmentation task using fixed-encoder weights from Ima- geNet and MoCo-V2. The segmentation outputs compared with both the original RGB image and the segmentation labels.
5.4 Fine-Grained Semantic Segmentation
Unlike AV+, this dataset is severely limited by the availability of fine-grained segmentation labels. There are 184 tiles, from 68 flights, this dataset that are split into training (70%), validation (15%), and test (15%). Again, use U-Net architecture with ResNet-18 encoder. For training, use a multi-class focal loss Lin et al. (2077) to account for the strong class imbalance.
Results are shown in Table 4 and sample output is shown in Figure 7 Results improve across the board when both the encoder and decoder are fine-tuned. Although less dramatic than the results seen on the AV+ classification and segmentation tasks, some improvement over ImageNet weights is seen using the MoCo-v2 framework with ResNet-18 backbone for fixed weights. As seen on the other tasks, when the entire network undergoes fine-tuning, the ImageNet and SSL weights, specifically MoCo-PixPro produce roughly the same performance on the downstream task. Additional per-class analysis is provided in the Supplemental. The ResNet-50 and Swin-T models performed relatively worse compared to the ResNet-18 models, which is unsurprising given the extremely small size of this dataset.
5.5 Land-Cover Classification on EuroSAT
We further prove pretraining on the AV+ dataset benefits the downstream task the broader remote sensing community. We conduct downstream classification experiments EuroSAT (Helber al. 2019). EuroSAT addresses the classification challenge of land use and land cover with images from Sentinel-2. It consists of 27,000 labeled images and 10 classes over 34 European countries. We use the splits protocol of train/val following the work of Neumann et al. 2019))
We freeze the pre-trained backbones and add a lincar layer to evaluate the learned representation in this classification task. Totally, the linear layer is tunned with 100 epochs using the Adam optimizer. The initial learning rate is set 0.001 and is divided by 10 at the 60th and 80th epochs.
12

---

Published in Transactions on Machine Learning Research (03/2023)
MoCo-V2 McCo-PixPro
TemCo
TemCo-PixPro
MoCo-V2 MoCo-PixPro
TemCo TemCo-PixPro
80
Chastication Accuracy of Resivet-1
70
76
Clocefication Accurncy
72
70
86
64
60
0.01
0.05
0.10
0.50
0.01
0.05
0.13
0.50
1,00
Percentage labeled dala
Percenlage of labeled dall
Figure 8: Ablation study on the pre-training size of data on different pre-training methods on two downst tasks. Solid lines represent accuracy from 3600 flights while dashed lines represent accuracy from 1200 flights. Left: results from non-linear probing downstream classification. Right: fine-tuning results entire networks for downstream AV+ segmentation.
The results shown the Table 3 compare weights pre-trained from AV+ against other baselines. We notice that MoCo-Va V2 and our proposed MoCo-PixPro achieve 1.21% and 6.25% higher accuracy compared with ImageNet'ss weights accordingly. These results confirm not only the effectiveness of pre-training AV+ but also AV+'s significant potential to generalize to the broader remote sensing field.
Table 5: Accuracy of the EuroSAT land-cover classification task using ResNee-18 Weights
Random
ImageNet
MoCo-V2 MoCo-PixPro
Accuracy (%) 63.21
86.32
87.53
89.97
5.6 Ablation Study: Number of Flights
We ResNet-18 backbone and basic MoCo-V2 for experiments. When the number of flights used for SSL is increased from 300 to 3600, observe stable improvement in the downst classification task under the non-linear probing setting; this gain is confirmed regardless of the fraction of the labeled dataset for tuning. See Supplemental: Additional Results for more detailed results.
This improvement is seen for all examined SSL methods Figure g when the raw dataset is increased from 1200 to 3600 flights and evaluated under non-lincar probing for classification and full-network fine-tuning for AV+ segmentation. Our SSL models' performance steadily grows as raw data size increases, suggesting that even more data may lead to even greater performance.
6 Conclusion
Large, high-quality datasets are opening tremendous new opportunities for computational agriculture, but they are extremely difficult obtain. As in other domains, remote sensing and earth observation data are marked by huge amounts of unlabeled data and relatively few annotations; leveraging the information in this unlabeled data, therefore, becomes critical task. In this work, contribute to the advancement of these efforts by relcasing the AV+ dataset, which contains annotated full-field imagery based on the original AV dataset Chiu el al. (2020b), supplemented by more than 3TB raw full-field images taken at different times in the season. The improved supervised component of the AV dataset will allow for greater flexibility in training and augmentation protocols and enable additional possible lines of study around long-range context and large-scale imagery. The raw unlabeled data will enable continued exploration in the self, semi, and weaks weakly kly supervised methods which have begun to benchmark here. This extension of an already important
13

---

Published in Transactions on Machine Learning Research (03/2023)
dataset in computational agriculture will open up many lines of research and investigation which benefit both the agriculture and computer vision communities.
Next, we conduct a thorough benchmark study on self-supervised pre-training methods based on contrastive learning, which captures the fine-grained, spatiotemporal nature this data. We analyze a classification formulation of the AV+ dataset under linear probing, non-linear probing, and fine-tuning. We also exam- ined segmentation tasks, which are often overlooked in remote sensing approaches, based on the original segmentation formulation of AV+ with a frozen and unfrozen encoder and an extremely small fine-grained segmentation task under the same formulations. Our benchmark study explores both traditional CNN ar- chitectures (ResNet-18 and ResNet-50) as well as the more recent Swin Transformer, which offers unique potentials for computer vision, but requires huge amounts of data to train.
Importantly, we incorporate the Pixel-to- Fixel-to-Provd Poopaaagttooonn Module, originally built in the SimCLR framework, into the MoCo-V2 framework, which allows for training on larger batch sizes. Our results show that this module is key for downstream segmentation and classification tasks, even though it was designed primarily for dense detection and segmentation tasks. As our dataset contains richer low-level, high-frequence fine-grained features than traditional natural imagery like COCO or ImageNet, this suggests that PPM is beneficial for learning dense, fine-grainee features in addition to dense label structure.
We further combine this module with a TemCo, a modification of SeCo, into a rich framework that captures the dense, spatiotemporal structure of our data. While this combined framework was not the highest- performing on the various task, it again may have been at a disadvantage since is a larger model and the number of steps was fixed for fair comparison. Additionally, extending how posilive samples are generated could prove beneficial. These improvements
the focus of future analysis.
Self-supervised methods will be crucial for unlocking opportunities in remote sensing, particularly for agri- culture, and this dataset release and benchmark study offer significant step in that direction.
References
Saleh Albelwi. Survey on self-supervised learning: auxiliary pretext tasks and contrastive learning methods in imaging. Entropy, 24(4):551, 2022.
Kumar Ayush, Burak Uakent, Chenlin Meng, Kunar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. Geography-ave self-supervised learning. arXiv preprint (rXic:2011.09980 2020.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual
information across views. arXiv preprint vrXiv:19060091. 2019.
Antonio Barrientos, Julian Colorado, Jaime del Cerro, Alexander Martinez, Claudio Rossi, David Sanz, and Joao Valente. Aerial remote sensing in agriculture: A practical approach to area coverage and path planning for fleets of mini aerial robots. Journal of Field Robotics, 28(5):667 -89, 2011.
Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML workshop on unsupervised and transfer learning, pp. 17 36. JMLR Workshop and Conference Proceedings, 2012.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0900 09888. 2020.
Qing Chang, Junran Peng, Lingxi Xie, Jiajun Sun, Haoran Yin, Qi Tian, and Zhaoxiang Zhang. Data: Domain-aware and task-aware self-supervieed learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0841-9850, 2022.
14

---

Published in Transactions on Machine Learning Research (03/2023)
Liang- Chig-Chihhhea Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the Euuropean con- ference on computer vision (ECCV), pp. 801-818, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0420 2020b.
Mang Tik Chiu, Xingqian Xu, Kai Wang. Jennifer Hobbs, Naira Hovakimyan, Thomas S Huang, and Honghui Shi. The 1st agriculture-vision challenge: Methods and results. In Proceedings of the IEEE/CVE Confer- ence on Computer Vision and Pattern Recognition Workshops, pp. 48-49, 2020a.
Mang Tik Chiu, Xingqian Xu, Yunchao Wei, Zilong Huang, Alexander G Schwing, Robert Brumner, Hrant Khachatrian, Hovnatan Karapelyan, Tvan Dozier, Greg Rose, el al. Agricalture-vision: A large aerial image database for agricultural pattern analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2828-28335 2020b.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understand- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
Rodrigo Caye Daudt, Bertr Le Saux, Alexandre Boulch, and Yann Gousseau. Urban change detection for multispectral earth observation using convolutional neural networks. IGARSS 2018-2011 IEFE Interna- tional Geoscience and Remote Sensing Symposium, pp. 2115 2118, 2018.
l1ke Demir, Krzysztof Koperski, David Lindenbaum, Guan Pang, Jing Huang, Saikat Basu, Forest Hughes, Devis Tuia, and Ramesh Raskar. Deepglobe 2018: A challenge to parse the earth through satellite images. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp.
172-181, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEFE Conference on Computer Vision and Pottern Recognition, pp. 248 255, 2009. doi: LLL09//VVR.2005
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprinl ar N8i0:1810004800 2018.
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 201-208. JMLR Workshop and Conference Proccedings, 2010.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Wimn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303 038,, 2010.
Min Feng and Yan Bai. A global land cover map produced through integrating multi-source datasets. Big Earth Data, 13):119-,, 2019.
Anatoly Gitelson, Yoram Kaufman, Robert Stark, and Don Rundquist. Novel algorithms for remote estimation of vegetation fraction. Remote sensing of Environment, 80(1):76-87, 2002.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya,
Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supevvised learning. arXiv preprint ar Xiv:2006.007333 2020.
15

---

Published in Transactions on Machine Learning Research (03/2023)
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognation, pp. 5356-5364, 2019.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '6)) volume 2, pp. 1735 1742. IEEE, 2006.
Tengda Han, Weidi Xie, and Andrew Zisserman. Memory-angmented dense predictive coding for video representation learning. In European conference on computer vision, pp. 312-329. Springer, 2020.
Sebastian Haug and Jörn Ostermann. A crop/weed field image dataset for the evaluation of computer vision based precision agriculture tasks. 2014.
Schastian Haug and Jörn Ostermann. A crop/weed field image dataset for the evaluation of computer vision based precision agriculture lasks. In European conference on compuler vision, pp. 105 116. Springer, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729 977399978, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, (17))2211-2226, 2019.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Confer- ence on Machine Learning, pp. 4182 4192. PMLR, 2020.
Joshua Kelcey and Arko Lucieer. Sensor correction of a 6-band multispectral imaging sensor for uav remote sensing. Remote sensing, 466):1462-1493, 2012.
Adnan Khan, Sarah AlBarri, and Muhammad Arslan Manzoor. Contrastive sele-supe learning: a survey on different architectures. In 2022 2nd International Conference on Artificial Intelligence (ICAT), pp. 1-6. IEEE, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.0 6980, 2014.
arviv preprint
Alex Krizhevsky, Ilya Sutskever, and Gcoffrey Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervisod representations. arXiv preprint arXiv:2005. 0rX1:22005.04666, 2020.
Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation. Advances in Newral Information Processing Systems, 34, 2021.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense object detection. CoRR, abs/1708.02002, 2017. URL http://aaxsv.orr/aa////0....00 wttp:/bb////00.000000
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021a.
16

---

Published in Transactions on Machine Learning Research (03/2023)
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint or arXim:106. 13230, 2021b.
Emmanuel Maggiori, Yuliya Tarabalka, Guillaume Charpiat, and Pierre Alliez. Can semantic labeling methods generalize any city? the inria acrial image labeling benchmark. In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 3226 3229. IEEE, 2017.
Oscar Mañas, Alexandre Lacoste, Xavier Giro-i Nieto, David Vazquez, and Pau Rodriguez. Scasonal contrast: Unsupervised pre-training from uncurated remole sensing data. arXiv preprinl arXiv:2103. 16607, 2021.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXie: 1011:39111,7811 2013.
A.A.O. Monteiro and A. von Wangenheim. Orthomosaic dataset of rgb aerial images for weed mapping. 2019.
http://www.lapp: ufsc. ufsc.
David J Mulla. Twenty five years of remote sensing in precision agriculture: Key advances and remaining knowledge gaps. Biosystems engineering, 114(4):358-377 371,, 2013.
Maxim Neumann, Andre Susano Pinto, Xiaohua Zhai, and Neil Houlsby, In-domain representation learning for remote sensing. arXiv preprint arXto:1911.0677 2019.
Alex Olsen, Dmiiry A Konovalov, Bronson Philippa, Peter Ridd, Jake C Wood, Jamie Johns, Wesley Banks, Benjamin Girgenti, Owen Kenny, James Whinney, et al. Deepweeds: A multiclass weed species image datasct for deep learning. Scientific reports, 9(1):1-12, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXi::1807.0S74 2018.
Anushree Ramanath, Saipreethi Muthusrinivasan, Yiquı Xie, Shashi Shekhar, and Bharathkumar Ra- machandra. Ndvi versus cmn features in deep learning for land cover clasification of aerial images. In IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium, pp. 6483 6486. IEEE, 2019.
David Rolnick, Priya Donti, Lynn H Kaack, Kelly Kochanski, Alexandre Lacoste, Kris Sankaran, An- drew Slavin Ross, Nikola Dupont, Natasha Jaques, Anna Waldman-Browt et al. Tackling climate change with machine learning. arXiv preprint arXiv:1906.05A4 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisee interven- lion, pp. 234 241. Springer, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 2222222,,1221111111 2015.
Santhosh K Seelan, Soizik Laguette, Grant M Casady, and George A Seielstad. Remote sensing applications for precision agriculture: A learning community approach. Remole sensing of encironment, 88(12)):157 169, 2003.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint Xix::11000 1556, 2014.
Suriya Singh, Anil Batra, Guan Pang, Lorenzo Torresani, Saikat Basu, Manohar Paluri, and CV Jawahar. Self-supervised feature learning for semantic sogmentation of overhead imagery. In BMVC, volume 1, pp. 4, 2018.
Leslie N. Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464-472, 2017. doi: 10. 10.1109/WACV.20 2015.
17

---

Published in Transactions on Machine Learning Research (03/2023)
Gencer Sumbul, Marcela Charfuelan, Begüim Demir, and Volker Markl. Bigearthnet: A large-scale benchmark archive for remote sensing image understanding. RSS 2019 2019 TEEE International Geoscience and Remote Sensing Symposium, pp. 5901- 5904, 2019. doi: N0.1199/IAARSS.20101 20t...2099..90
Shu Tian; Lihong Kang, Xiangwei Xing, Zhou Li, Liang Zhao, Chunzhuo Fan, and Ye Zhang. Siamese graph embedding network for object detection in remote sensing images. IEEE Geoscience and Remote Sensing Lellers, 18(4):002 606, 2020.
Gabricl Tseng, Ivan Zvonkov, Catherine Lilian Nakalembe, and Hannah Kerner. Cropharvest: A global dataset for erop-type classification. In Thirby-fiihh Conference on Newral Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Cees J Van Westen. Remote sensing and gis for natural hazards assessment and disaster risk management. Treatise on geomorphology, 3:259-295 2013.
Jing Wu, Ran Tao, Pan Zhao, Nicolas F. Martin, and Naira Hovakimyan. Optimizing nitrogen management with deep reinforcement learning and crop simulations. In Proceedings of the IEEE/CVI CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp.
June 2022.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pp. 3733 3742, 2018.
Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xiaoqiang Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 35(7):3965-330 3981, 2017.
Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Sele-supe learning with swin transformers. arXie preprint arXiv:2105.0455: 2021a.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceediiggs of the TEFE/CV /CV Conference on Computer Vision and Pattern Recognition, pp. 16684-16693, 2021b.
Jun Yang, Peng Gong, Rong Fu, Minghua Zhang, Jingming Chen, Shunlin Liang, Bing Xu, Jiancheng Shi, and Robert Dickinson. The role of satellite remote sensing in climate change studies. Nature climate change, 3(10):875 883, 2013.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017.
Adrian Ziegler and Yuki M Asano. Self-supervised learning of object parts for semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognation, pp. 14502 14511, 2022.
18


Reviews:
[{"review_id": "8mAQGYqZCd", "reviewer": "Reviewer_TPh3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#Strength#\n\nS1. The paper considers both classification and segmentation tasks and considers an even fine-grained segmentation task.\n\nS2. The paper considers multiple neural network architectures and multiple pre-training strategies. \n\n#Weakness#\n\n**W1. Writing needs significant improvement.** There are many typos and grammatical errors, such as “in symmetric form”, “are follow”, “performs best”, “AG+ or AV+” just to point out a few. I would strongly recommend that the authors run a Grammarly check. The citation format also can be improved --- please use \\citep, \\citet, \\cite appropriately. Also, there are some logically inconsistent sentences, such as: *\"Many approaches originally developed for natural images work well on remote sensing imagery with only minimal modification, although this is not guaranteed due to the large domain gap. Additionally, they may fail to exploit the unique structure of earth observation data such as …\"* The writing issue makes it a bit hard to read the paper.\n\n**W2. The background information, approach, and experimental setup are not well organized and presented.**\n\nW2.1. Figure 2 is not self-contented. There are no clear meanings for the use of colors. Also, the same block shapes sometimes refer to parameters or to losses, making the figure a bit confusing. \n\nW2.2. The loss in sect. 4.2 is a bit hard to grasp. First, $y_i$ is defined but not used. Second, how is the pair $(i, j)$ defined? Are $i$ and $j$ the same pixels but at different pixel locations due to augmentation?\n\nW2.3. It is a bit hard to understand the usefulness/goal of temporal contrast. Do the authors expect features across different times to be invariant? However, at different times, the patterns/labels to be recognized may change.\n\nW2.4. Fig 3 and Figures 4 and 5 are not in the same style.\n\nW2.5. Perhaps the severest issue of the paper is the lack of a clear description of the experimental setups. **First**, given that AV+ provides the full-field images, are the pre-trained and downstream models trained upon tiles or the full-field images? What are their sizes (height x width)? How many full-field labeled images are there? (94986, or more)? How are these images/tiles separated into train/test/validation? Is there a guarantee that nearby images (or images at the same geo-location) won’t be included in both train and test sets? Is the unlabeled set (2019-2020) collected at the same locations as the labeled set (2017-2019)? Logically, it might be a bit strange to use earlier images for testing/downstream tasks, and later images for pre-training. **Second**, there is no detailed description of the training and test protocols. **Third**, the difference between the \"normal\" segmentation data in AV+ and those in the \"fine-grained\" set is not clear. Does the \"higher resolution\" refer to the input image or just the output labeling? Also, how are the ground-truth labels obtained? If it is by humans, is there a carefully-designed way to guarantee quality? **Finally**, suppose the numbers of categories in normal fine-grained and fine-grained segmentation are the same (9 categories), it is unclear to me why only the latter encounters label imbalance and needs the focal loss.\n\n**W3. Lack of contributions and insights.**\n\nW3.1. To me, the extension of MoCo-V2 with Seasonal Contrast and PPM seems to be quite straightforward (i.e., there seem to be no difficulties in combining them). Thus, this contribution is quite marginal. \n\nW3.2. I would hope to see an analysis of how much the NIR channel contributes to the final performance, but there is no such analysis.\n\nW3.3. In sect. 5.4, there is no further discussion and analysis on why the ImageNet pre-trained model leads to the best performance. Also, the argument on larger models *\"The ResNet-50 and Swin-T models performed relatively worse compared to the ResNet-18 models which is unsurprising given the extremely small size of this dataset.\"* seems not convincing --- the authors already use a large dataset for pre-training. If the issue is in the decoder, the authors may consider a supervised pre-training step for the decoder using the segmentation data in AV+.\n\nW3.4. Results in Table 2 seem to be not comparable – different training strategies and model architectures are used. Also, did the original AV paper use the NIR channel?\n\nW3.5. There is no further discussion and comparison on how the full-field images help the classification or segmentation. As the release of the full-field images seems to be a contribution, it would be great to see such a comparison.\n\nW3.6. Except for showing that PPM improves the pre-training quality, there seem to be limited insights from this paper. I would hope to see some more discussions on why the temporal contrast does not consistently improve the performance. I would also want to see more discussions on how this paper improves scientific understanding, e.g., in agriculture. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper extends the original Agriculture-Vision (AV) dataset with full-field labeled images, full-field unlabeled images for pre-training, and finer-grained labeled images for segmentation. (contribution on datasets) The authors then evaluate different pre-training strategies (together with different model architectures) on the new dataset AV+. (contribution on benchmarking).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the above weaknesses.\n\nAdditionally, I strongly encourage the authors to show some images that contain all nine patterns and clearly illustrate the difference between the segmentation data in AV+ and the fine-grained segmentation data. I also strongly encourage the authors to create a table that clearly summarizes the experimental setups.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NgT3rPpRPa", "reviewer": "Reviewer_VzZZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The proposed datasets including a pre-training one are novel.\n- The experiments are sufficient and convincing.\n- The idea is easy to understand.\n- The paper's organization is clear.\n\n\n**Weaknesses**\n1. At the human knowledge level, what are the key agricultural patterns, which are concerned with the survival and development of agriculture? Which patterns are reflected in the proposed datasets? To what extent do they help agriculture?\n\n2. The summary of main contributions should be more concise, compact, and precise.\n\n3. The motivations for some contributions are not clear. For example, why does this paper focus on momentum contrastive learning methods rather than other types such as performing pretext tasks and contrasting cluster assignments [a,b,c,d]? Why does this paper incorporate PPM into the MoCo-V2 and SeCo frameworks? How does PPM work? Does such a design achieve the SOTA performance over existing self-supervised learning methods on the proposed datasets? In a word, the introduction section should be self-contained.\n\n[a] Albelwi, S. Survey on Self-Supervised Learning: Auxiliary Pretext Tasks and Contrastive Learning Methods in Imaging. Entropy 2022, 24, 551. https://doi.org/10.3390/e24040551.\n\n[b] Qing Chang, Junran Peng, Lingxi Xie, Jiajun Sun, Haoran Yin, Qi Tian, Zhaoxiang Zhang; DATA: Domain-Aware and Task-Aware Self-Supervised Learning; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 9841-9850.\n\n[c] Adrian Ziegler, Yuki M. Asano; Self-Supervised Learning of Object Parts for Semantic Segmentation; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 14502-14511.\n\n[d] A. Khan, S. AlBarri and M. A. Manzoor, \"Contrastive Self-Supervised Learning: A Survey on Different Architectures,\" 2022 2nd International Conference on Artificial Intelligence (ICAI), Islamabad, Pakistan, 2022, pp. 1-6, doi: 10.1109/ICAI55435.2022.9773725.\n\n4. Is it enough for downstream transferring to pre-train the model on only 3,600 images? As we know, the ImageNet dataset has 1.28M natural images. Thus, a problem is raised naturally: how does the pre-training data quantity affect the downstream classification or segmentation performance? Why don't the authors collect more full-field images for pre-training?\n\n5. At the pre-training stage, the pixel-level loss is computed between the smoothed q_i^s from PPM and k_j. Without any annotations, how can it be sure that pixels i and j are a positive pair?\n\n6. The data augmentation operations used in MoCo-V2 are tailored for natural images. Would they be fit for remote sensing imagery? How does remote sensing imagery differ from natural images in terms of appearance and structure? I wonder whether there are data augmentations designed for the characteristics of remote sensing imagery.\n\n7. In Table 1, with ResNet as the backbone and fine-tuning as the downstream training strategy, MoCo-PixPro performs worse than MoCo-V2, why does PPM yield a negative effect? \n\n8. The line number should be shown for ease of reference!\n\n9. Grammar and symbol errors. The authors should carefully proofread the paper and revise the errors. For example, at line 5 of paragraph 1 in Sec. 1, 'exists' -> 'exist'; at line 8 of paragraph 3 in Sec. 1, 'agriculture dataset' -> 'agriculture datasets'; at line 9 of paragraph 3 in Sec. 1, 'which is' -> 'which are'; at last line of paragraph 2 in Sec 3.3, 'resemble' -> 'resembles'; at line 7 of paragraph 1 in Sec. 4.1, 'q and k' -> 'k+ and k-'; at line 5 under Eq. (3), 'features maps' -> 'feature maps'; at line 1 of paragraph 1 in Sec 5.1, 'follow' -> 'following'. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To push forward agricultural pattern analysis, this paper improves the existing Agriculture-Vision dataset by including raw, full-field imagery. Then, this paper extends this dataset by releasing 3,600 large, high-resolution, full-field, RGB, and near-infrared images, which enable unsupervised pre-training. Next, this paper integrates the Pixel-to-Propagation Module from the SimCLR framework into different contrastive learning approaches for improvement. Finally, this paper benchmarks different self-supervised pre-training methods based on momentum contrastive learning on downstream classification and segmentation tasks. Here, CNN and Swin Transformer are used as backbones.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses.", "broader_impact_concerns": "The work has no obvious ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UKqoWdtrAl", "reviewer": "Reviewer_osPe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The proposed dataset augments the original Agriculture-Vision dataset, which includes more than 3600 full-field aerial images collected across 4 years and a small subset with fine-grained segmentation annotations. It is valuable for developing new pre-training methods in remote sensing and fine-grained agricultral pattern analysis. \n\n+ The paper benchmarked a series of MoCo-based pretraining methods, including four variants on two types of network backbones. In particular, it integrates the PPM and/or temporal contrast with MoCo, which provides a detailed study on self-supervised learning strategies from image, pixel and temporal perspective in this application domain.\n\n+ The experimental comparison with the original Agriculture-Vision model demonstrates the superior performance of the methods with pre-training on the segmetnation task. \n\nWeaknesses:\n\n1. Related work: \n- Other self-supervised learning techniques have also been explored for semantic segmentation, which was not discussed in this work. While this paper aims to build a generic pre-training model, adding those works could provide a more complete perspective on how the data can be used.  E.g.:\nSingh, Suriya, et al. \"Self-Supervised Feature Learning for Semantic Segmentation of Overhead Imagery.\" BMVC. Vol. 1. No. 2. 2018.\nZiegler, Adrian, and Yuki M. Asano. \"Self-Supervised Learning of Object Parts for Semantic Segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n- The citation format seems incorrect. For example, in Sec 2.1,  \"Specifically pertinent to the current work, Z. Xie et al. (2021) Xie et al. (2021b)  extended the SimCLR framework...\". This looks redundant. \n\n2. The proposed SSL study is lacking in several aspects: \n- Geo-location information is missing from the representation learning, which seems counter-intuitive. It seems the geography information places an important role in agriculture pattern analysis. \n- There is no object-level recognition in the downstream tasks. How well the learned representation would perform on the localization of object instances? \n- There is no few-shot learning study on the pre-trained models. What if the labeled data are reduced to a very small amount? \n\n3. Some experimental details are missing. For example, in Sec 5.3.2., the compairson with the supervised method lacks details: Table 2 should include the model sizes for a fair comparison. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an extended version of the aerial image dataset, Agriculture-Vision, for agricultural pattern classification and segmentation, and builds a benchmark for self-supervised pre-training methods for the agriculture-related downstream tasks. In particular, this work evaluates two backbone networks (ResNet and Swin Transformer) and four different MoCo-based pre-training methods, demontrating their effectiveness on four downstream tasks, including agricultural pattern classification, (fine-grained) semantic segmentation and land-cover classification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the three concerns in the weaknesses. ", "broader_impact_concerns": "I have no concerns on the ethical implications. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RSiX6dJHLP", "reviewer": "Reviewer_GJ9C", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* [S1] As mentioned in the paper, the large-scale benchmark release will unlock opportunities in remote sensing, particularly for agriculture in the community.\n* [S2] Experimental results in Table 1 demonstrate the scale of the dataset and its value for self-supervised representation learning. The ResNet-based models are inferior to Swin-T, as the latter is more data-hungry. In addition, the performance of all models has not been saturated (~30% mIoU), which offers future opportunities for self-supervised learning on this benchmark.\n\n\nWeaknesses\n* [W1] The motivations behind Section 4.3-4.4 are very unclear. The reviewer would like to see more details on the temporal information. It is important to include figures illustrating the temporal information and explain how this information can be useful for downstream tasks, especially for an audience who is interested in using AV+ dataset but doesn’t have related domain knowledge. Without knowing this information on the dataset, it is hard to justify the usage of “temporal contrast” and the extension.\n* [W2] The writing of section 5.4 is a bit unclear. First, the reviewer cannot tell if “this dataset” is part of AV+ release or not from the first sentence. Second, since the dataset is small, the results on the dataset do not seem to be conclusive. Please elaborate on the “fine-grained semantic segmentation” dataset and compare the dataset with existing ones in the field.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new release that extends the Agriculture-Vision dataset. Specifically, the proposed version contains more than 3,600 full-field images to encourage self-supervised representation learning. Further, this paper benchmarks various self-supervised methods on this new dataset with extensive studies using different backbones (e.g., ResNet and SwinT) and contrastive learning methods (e.g., Pixel-to-Propagation, MoCo-V2, and Seasonal Contrast). Finally, this paper adapts the SeCo to handle the dense and spatiotemporal structure of the image data captured over time.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Address [W1]: Please clarify the temporal information and include samples captured over a coarse of the 5-month period (in order) of the same region with labels. Justify the \"temporal contrast\" and its extension is legitimate given the temporal information of the dataset.\n\nAddress [W2]: Clarify the size and statistics (e.g., how many label categories) of the fine-grained segmentation dataset. Justify the results are conclusive given the small dataset or revise this section if the results are not conclusive. Discuss the alternative datasets for fine-grained segmentation in this field. If possible, show results on the alternative datasets.", "broader_impact_concerns": "I am concerned if researchers in general remote sensing will be interested in using the dataset, as the domain-specific knowledge has not been explained well in the manuscript. It would be good to revise the related paragraph and incorporate this information. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper contributes a dataset that extends the existing Agriculture-Vision dataset by including 3,600 large, high-resolution, full-field, RGB, and near-infrared images. The paper then studies pre-training methods on the dataset, focusing on the momentum contrast framework, and integrates the Pixel-to-Propagation Module into it. This paper benchmarks these self-supervised pre-training methods on downstream classification and segmentation tasks, showing some useful insights and future research directions.

Four reviewers carefully review the paper. While they generally accept the paper contributions, they also raise critical concerns. A few common ones include 1) less clear writing, including some confusing math notations and typos, 2) the less clear motivation and description of temporal information and contrast in the dataset, 3) why to choose momentum contrastive learning methods for the study, rather than other self-supervised learning methods. The authors responded and revised the paper that makes the paper clearer. Reviewers are generally satisfied with these common issues.

In addition, Reviewer TPh3 raised issues about style inconsistency among a few figures, unclear description of the experimental setups, training/test protocols, etc, and less significant contributions and insights from the paper. The authors have carefully responded to these issues and revised the paper accordingly. The reviewer appreciates most of these responses.  

After the discussion phase, two of the reviewers suggest acceptance while the others suggest not. AE follows up the discussions and would like to accept the paper given that 1) the releasing of this dataset is valuable for agricultural research, 2) while only momentum contrast based methods are studied, the studies are self consistent and generalizable to other self-training methods, 3) the paper and the dataset would advance the field by bringing features (imagery data, temporal contrast, fine-grained classification, etc) new to the community. 

In the final version, the authors are strongly suggested to follow the suggestions from the reviewer and make writing of the data collection and training/testing protocol clear and self-sufficient, rather than to ask the readers to refer to the previous literature. Please also revise the format of citation and correct typos throughout the paper,
Decision: Accept (poster)