IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 6, NO. 4, OCTOBER 2021

7815

Leveraging Metadata in Representation Learning
With Georeferenced Seafloor Imagery
Takaki Yamada , Miquel Massot-Campos , Adam Prügel-Bennett , Stefan B. Williams ,Oscar Pizarro , and Blair Thornton

Abstract—Camera equipped Autonomous Underwater Vehicles(AUVs) are now routinely used in seafloor surveys. Obtaining
effective representations from the images they collect can enable
perception-aware robotic exploration such as information-gainguided path planning and target-driven visual navigation. This letter develops a novel self-supervised representation learning method
for seafloor images collected by AUVs. The method allows deeplearning convolutional autoencoders to leverage multiple sources of
metadata to regularise their learning, prioritising features observedin images that can be correlated with patterns in their metadata.
The impact of the proposed regularisation is examined on a dataset
consisting of more than 30 k colour seafloor images gathered by an
AUV off the coast of Tasmania. The metadata used to regulariselearning in this dataset consists of the horizontal location anddepth of the observed seafloor. The results show that including
metadata in self-supervised representation learning can increase
image classification accuracy by up to 15% and never degrades
learning performance. We show how effective representation learning can be applied to achieve class balanced representative image
identification for summarised understanding of imbalanced class
distributions in an unsupervised way.
Index Terms—Marine robotics, representation learning, visual
learning, computer vision, metadata.

I. INTRODUCTIONMAGES gathered by camera equipped Autonomous Underwater Vehicles (AUVs) are now used in a wide range of
seafloor survey applications. The captured images are used tocharacterise seafloor scenes where both manual and automaticI

Manuscript received February 23, 2021; accepted July 8, 2021. Date of publication August 4, 2021; date of current version August 17, 2021. This letter wasrecommended for publication by Associate Editor N. Mahmoudian and Editor P.
Pounds upon evaluation of the reviewers’ comments. This work was supported
in part by the U.K. Natural Environment Research Council’s Oceanids Biocam
Project NE/P020887/1, in part by Australian Research Council’s AutomatedBenthic Understanding Discovery Project DP190103914, and in part by EU
Horizon 2020 TechOceanS Grant ID 101000858.
Takaki Yamada, Miquel Massot-Campos, and Adam Prügel-Bennett are with
the Centre for In Situ and Remote Intelligent Sensing, Faculty of Engineering,and Physical Science, University of Southampton, Southampton SO17 1BJ,
U.K. (e-mail: t.yamada@soton.ac.uk; miquel.massot-campos@soton.ac.uk;
apb@ecs.soton.ac.uk).
Stefan B. Williams and Oscar Pizarro are with the Australian Centre for Field
Robotics, The University of Sydney, Camperdown, NSW 2006, Australia (email: stefan.williams@sydney.edu.au; o.pizarro@acfr.usyd.edu.au).
Blair Thornton is with the Centre for In Situ and Remote Intelligent Sensing,
Faculty of Engineering and Physical Science, University of Southampton,
Southampton SO17 1BJ, U.K, and also with the Institute of Industrial Science, The University of Tokyo, Bunkyo City, Tokyo 113-8654, Japan (e-mail:b.thornton@soton.ac.uk).
Digital Object Identifier 10.1109/LRA.2021.3101881

methods are used for interpretation. Typical AUV missions will
gather tens to hundreds of thousands of images during a single
dive, where the high resolution and large redundancy of imagery
pose a significant challenge for automated interpretation. In order to take full advantage of images for robotic applications, it isnecessary to obtain compact representations that efficiently preserve the most valuable information in the original images. Oncethese are generated, algorithmic analysis can be performed withlow latency using relatively limited computational resources.
Examples of robotic applications that could be facilitated using
image representation include information-gain-aware path planning for representative surveys [1], [2], target-aware seek andsample missions [3], curiosity-driven exploratory surveys [4],and real-time habitat inference [5], [6].
The aim of this letter is to develop a self-supervised learning
method that can use metadata gathered with seafloor imagery
to efficiently generate low-dimensional latent representation
spaces that are useful for image interpretation. Effective lowdimensional representations form the basis of semantic interpretation, where classification, clustering, and content based
retrieval are examples of tasks that can be readily applied toachieve efficient understanding of underwater scenes. Fig. 1
illustrates a typical AUV survey scenario. Data is often gathered over multiple dives, where ships transport AUVs betweensites between their dives. These locations can be separated bydistances far larger than that traversable by an individual AUV.
Observations typically cover spatial extents several orders ofmagnitude larger than the footprint of a single image frame,
which typically have edge lengths of a few metres, and spana wide range of seafloor depths. Habitats and substrates varyover spatial scales larger than each image and exhibit patterns
with depth, especially in shallow water due to the influenceof sunlight. Therefore, images taken close to each other, or
separated but with similar depths, are more likely to share visualcharacteristics than would otherwise be the case. To leverage thisinformation, we implement our metadata regularised learning
method using horizontal location and depth information. A key
advantage of this approach is that regularisation can be applied
to data gathered in remote locations during different dives basedon depth information. The novel contributions of this work are:r Development of a regularisation method that leverages
metadata when training deep-learning Convolutional Neural Network (CNN) based autoencoders for efficient latent
representation of seafloor imagery.

2377-3766 © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: University of Tokyo. Downloaded on April 11,2022 at 10:16:28 UTC from IEEE Xplore. Restrictions apply.

7816

IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 6, NO. 4, OCTOBER 2021

Fig. 1. Overview of the proposed representation learning method for a typical multi-deployment AUV seafloor imaging survey. The method regularises therepresentation space of an autoencoder by embedding images taken at similar horizontal locations, or separated but with similar depths, in nearby regions of thelatent representation space. This is achieved by minimising the Kullback-Leibler divergence between the affinity matrix of the image latent representation withhorizontal location metadata using the loss function Lloc , and with depth metadata using the loss function Ldep . These are optimised together with the autoencoder
reconstruction loss Lrec to regularise the latent representation space according to these metadata.

r Implementation of the proposed method where an
AlexNet [7] based autoencoder is trained on seafloor images, regularised by a loss function that introduces domain
relevant assumptions on georeference (horizontal locationand depth) information.
r Performance validation on a dataset gathered off the coastof Tasmania that consists of more than 30 k seafloor images
and 2.2 k human annotations, taken over six AUV dives
between depths of 28 and 96 m, and demonstration of an
application to unsupervised representative image selection
to generate semantic summaries of the observations.

II. BACKGROUND
Seafloor habitats and substrates can be identified by uniquepatterns in their appearance, and various machine learningtechniques have been applied to automate image interpretation.
These can be broadly split into studies that use feature engineering, where descriptors are manually chosen or tuned by humanexperts, and representation learning, where descriptors are directly learnt from the data. In both cases, the reduced dimensionsof the representations allow for more effective identification ofpatterns in the data.

Manually engineered feature descriptors have been investigated by several groups for efficient image representation [5],[6], [8]–[11]. In [9], [11], colour-based descriptors were designed based on prior knowledge of targets that are of specificscientific interest. Generic descriptors such as Local Binary
Patterns (LBP) [12] and Sparse Coding Spatial Pyramid Matching (ScSPM) [13] have also been applied to identify spatiallyinvariant patterns that appear at different scales within images ofthe seafloor [5], [6]. In [10], accumulated histograms of oriented
gradients from image keypoints were used to describe seafloor
images for the purpose of clustering and anomaly detection.
However, these types of descriptor often require manual tuningof parameters, or feature engineering, to effectively describe thedatasets they are applied to.
CNNs avoid the need for feature engineering by learning
the latent representations needed to best describe the datasets
they are applied to. This is typically achieved by using labels
generated by humans experts to supervise CNN training, which
simultaneously optimises the latent representations and classboundaries to best describe the patterns of interest in a training
dataset. In [14], the ResNet [15] deep-learning CNN was trained
to distinguish between nine different classes of coral in a seafloor
image dataset, demonstrating higher classification resolution
than traditional feature engineering based methods. However,

Authorized licensed use limited to: University of Tokyo. Downloaded on April 11,2022 at 10:16:28 UTC from IEEE Xplore. Restrictions apply.

YAMADA et al.: LEVERAGING METADATA IN REPRESENTATION LEARNING WITH GEOREFERENCED SEAFLOOR IMAGERY

the need for large volumes of annotated images to supervise
CNN learning limits wide scale use in marine applications since
generic training datasets do not exist.
An alternative approach to train CNNs is to use selfsupervised learning techniques. In domains where continuity
exists between the samples in a dataset, this continuity canbe used to help regularise representation learning without theneed for direct human supervision. In natural language processing, Word2vec [16] and GloVe [17] leverage the assumptionthat words found in similar contexts are likely to have similar
meanings. This continuity was used to generate continuous
representations of different word nuances. For image processing
applications, Tile2Vec [18] extends the assumption to spatiallydistributed data, demonstrating its effectiveness for satellite
image interpretation. In [19], we developed a Location Guided
Autoencoder (LGA) that regularises autoencoder learning using
horizontal geo-location information for efficient clustering andcontent-based retrieval of seafloor imagery. In [20], a similarassumption is introduced for CNN-based coral detection fromseafloor imagery, where object tracking results in sequential
frames are used for semi-supervised training.
The method developed in this letter advances the state of theart for seafloor image interpretation. First, the use of metadata
is advanced by incorporating depth information in parallel tohorizontal location information for learning regularisation. Thisis significant as even though our previous LGA method usedhorizontal geo-location to regularise learning [19], this method
cannot regularise learning across large horizontal spatial discontinuities in observation, as is often seen between different AUV
dives. Additionally, although the effectiveness of the method
when applied to dense survey trajectories that fully cover a 2D
region of the seafloor has been demonstrated, it is not clear
how effective the method is for sparse trajectories. Dense survey
trajectories guarantee that each image has many other images in
its neighbourhood that it can be paired with to regularise learning. However, sparse trajectories are often used when surveying
larger regions of the seafloor, and under these conditions only asmall number of neighbourhood image pairs are available, which
potentially limits the effectiveness of the horizontal location
based regularisation. In contrast, depth information can provide alarge number pairings for sparse surveys and regularise learning
across different dives if depth related distribution patterns exist.
Next, we apply contrastive learning methods to improve theregularisation effect of the metadata, where this is the first time
contrastive learning has been applied to seafloor imagery. We
demonstrate these concepts on a dataset that consists of seafloor
imagery gathered over 6 AUV dives, with observations that aresparsely distributed over a 1.6 × 1.7 km region spanning a depth
range of 28 to 96 m.
III. METADATA REGULARISED AUTOENCODER
A. General Concept
An autoencoder consists of an encoder f (·) and a decoderg(·). The encoder f (·) maps a set of seafloor images x toa lower-dimensional tensors h (h = f (x)), and the decoderg(·) reconstructs the images xrec from h (xrec =g(h)) so that7817

the reconstructed images become as similar as possible to theoriginal images. The optimisation
 minimises the mean squared
error loss function Lrec = n1 n xrec − x2 , where n is thetotal number of images. Here h can be regarded as reasonable
latent representations of x since they preserve key information inx so that xrec can be reconstructed properly. The key advantage
of an autoencoder is that the encoder f (·) can be trained in aself-supervised manner, where only the input images are usedand no additional human annotations are needed. To incorporate
metadata into autoencoder training, we minimise a loss functionof the following form:
Lall = Lrec +

λ m Lm .

(1)

m is an index for each type of metadata used for learning
regularisation, where these can be any number of continuous
scalar or vector quantities that can be associated with the images.
Lm is the loss function that regularise autoencoder training basedon the values of metadata m. λm is a hyperparameter used tobalance the loss contributions.
B. Implementation for Georeferenced Imagery
AUVs typically measure their horizontal location, depth andaltitude for basic navigational functionality. This metadata canbe leveraged to regularise autoencoder training by formulating(1) as follows:
Lall = Lrec + λloc Lloc + λdep Ldep ,(2)

where Lloc is the loss function for the horizontal location based
regularisation, Ldep is for the depth based regularisation, λloc
and λdep are hyperparameters to balance their relative contributions. In our implementation, AlexNet [7] and its inverted
architecture are used as the encoder and decoder, respectively,
where any type of neural network can be used to constructautoencoder in a similar way. Our previous LGA method [19]can be regarded as a specific case of eq. (1), where only Llocand λloc are used.
1) Vector Based Regularisation: The horizontal location loss
Lloc is introduced to regularise autoencoder training following
the assumption that two images captured within a close distancetend to look more similar than two that are far away. In representation learning, if two images look similar and potentiallybelong to the same class, their latent representations should belocated within a close distance in the latent space. In order tomake the distribution of latent representations h reflect the 2D
horizontal location vector y where the images x are taken, we
introduce a loss function that has a similar structure to the loss
function of t-SNE [21]. In t-SNE, original high-dimensional
data xorg is embedded into a 2D or 3D space xemb so that datawith close relative distances in the original space are represented
with high probability in the embedded space. In our problem, y,which controls the distribution in the latent space corresponds
to xorg , and the latent representations h corresponds to xemb .
Following the t-SNE loss function, the probability pij , which is
proportional to the distance between y i and y j , is defined for

Authorized licensed use limited to: University of Tokyo. Downloaded on April 11,2022 at 10:16:28 UTC from IEEE Xplore. Restrictions apply.

7818

i=j as:IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 6, NO. 4, OCTOBER 2021


 
2
2
exp − y i − y j  /2σloc,
pj|i = 
2
2
exp
−
y
−
y

/2σ
i
k
k=i
loc

(3)

pj|i + pi|j,(4)2nwhere pij =0 when i=j, σloc is a normalising factor for y. The
probability qij is derived from h, and is optimised based on pij .
For qij when i=j, it is defined by the Student’s t-distribution as:
−1
1 + hi − hj 2(5)
qij = 

−1 ,
2
1
+
h
−
h

k
l
k=l
pij =

where qij =0 for i=j.
By defining the affinity matrices P and Q with pij and qij as
their elements, the horizontal location loss Lloc is defined as the
Kullback-Leibler (KL) divergence of P from Q:pij
pij log.
(6)
Lloc = KL(P Q) =
qij
i=j

Minimising Lloc forces Q to approach P , which embeds thecorrelation between the image representations and the horizontal
location metadata into the latent representation. Eq. (3)–(6) are
implemented in a similar way to the loss function of t-SNE,
where y is used to derive the target probabilistic distribution
instead of xorg , and h is optimised instead of xemb .
2) Scalar Based Regularisation: The depth loss Ldep can beformulated in a similar way to the horizontal location loss Lloc
defined earlier. Given that the seafloor depth where an image xi
is captured is a scalar value di , the probability rij is defined to
be proportional to the difference between di and dj where theobservations are made:2exp −(di − dj )2 /2σdep,
rj|i = (7)22k=i exp −(di − dk ) /2σdep
rj|i + ri|j,(8)2nwhere rij =0 when i=j. σdep is a normalising factor. The depth
loss is formulated as the KL divergence Ldep =KL(RQ), where
R is the affinity matrix with elements rij .
3) Generalised Regularisation Behaviour: An importantcharacteristic of the proposed method is that multiple regularisation methods can be applied without risk of significantly
degrading performance. As elements in the affinity matrices(e.g. P and R), become further apart in the metadata space(i.e. the distance between yi and yj or di and dj increases),
the values of pij or rij become less sensitive to the separating
distance. Furthermore, since the t-distribution used in this work
is heavy-tailed compared to Gaussian distributions, it avoids the“crowding problem” that can occur when high-dimensional data
is embedded into a lower-dimensional space when generating
a t-SNE. This is preferable to avoid over-regularisation by the
metadata, since pairs of images that are far apart are less strongly
constrained by the regularisation and can be flexibly embeddedrij =in the latent space. Since the loss function only loosely constrains
autoencoder training based on probabilistic distributions, it isinherently robust to over-fitting metadata. Furthermore, if the
training process finds a particular type of metadata to have little
correlation with the appearance of images, it gets automaticallyignored, and where a particular type of metadata is found to havea strong correlation with image appearance it gets increasingly
prioritised. This self-regulating characteristic is important insituations where many different types of metadata can be applied
as the method can automatically prioritise the most significant
metadata and mitigate any negative impact without additional
human input or tuning.
Here P and R are formulated for y and d, which are
2D (latitude-longitude) vectors and scalar values, respectively.
However, the proposed loss function can be implemented forany combination of vector or scalar metadata where the similarity between its values can be defined. This is important as itallows the proposed concept of metadata based regularisation to
be readily applied to different types of samples (e.g. seafloor
imagery, water column microscopy) and available metadata(e.g. acoustic back-scatter intensity, terrain rugosity, seawatertemperature, pH) depending on the configuration of the data
gathering platforms.
C. Mini-Batch Sampling and Contrastive Learning
Ideally, Lloc and Ldep would be derived from all the samples
in a dataset (i.e. n samples) so that they are globally optimised.
However, due to computational limitations, mini-batch gradientdescent is used for the simultaneous optimisation of Lrec , Lloc
and Ldep . The number of images considered at each iteration
is limited to a mini-batch size n∗ , where a strategy is needed
to avoid over-fitting to local minima in Lloc or Ldep whensampling n∗ images. Since the regularisation effect is diminishedas the number of horizontal location and depth neighbourhood
pairs reduces, we introduce a sampling method that balances
the number of images that are nearby and far away in each
metadata space. First, two images are randomly selected at each
iteration. Next n∗ /3 images are selected from the first image’s
horizontal location neighbourhood, and another n∗ /3 images are
selected from the second image’s depth neighbourhood, and thefinal n∗ /3 images are randomly selected from the whole dataset
in accordance with the principles of triplet loss contrastive
learning demonstrated in [22]. This ensures a large variety is
maintained in the values of the affinity matrices P and R,
which prevents over-regularisation and allows similar images
and dissimilar images to be evenly considered at each batch
iteration.
IV. EXPERIMENT
A. Dataset
The proposed method is applied to seafloor imagery obtained
off the east coast of Tasmania [23]. Analysis is performed on
32 097 seafloor images taken by the Australian Centre for
Field Robotic’s Sirius AUV from an altitude of 2 m. The data
analysed here was gathered over six dives sparsely covering a

Authorized licensed use limited to: University of Tokyo. Downloaded on April 11,2022 at 10:16:28 UTC from IEEE Xplore. Restrictions apply.

YAMADA et al.: LEVERAGING METADATA IN REPRESENTATION LEARNING WITH GEOREFERENCED SEAFLOOR IMAGERYTABLE ITASMANIA DATASET DESCRIPTION

7819

and so depth regularisation can be performed as long as thesesensors are properly calibrated.
B. Autoencoder Training

1.6 × 1.7 km region of the seafloor between 28 and 96 m depth.
Details of the survey are given in Table I.
The images show various habitat and substrate distributions,
including kelp (A), a registered essential ocean variable, androcky reefs (B) - (E), which can form habitats for various
conservation targets such as coral and sponges [24]. The original
resolution of the images is 1360 × 1024. Each image in thedataset is re-scaled to a resolution of 2 mm/pixel based on thecamera field of view (FoV) and imaging altitude. The centre
227 × 227 of each image is used in the analysis. The average
distance between adjacent images is approximately 0.5 m andso the overlap between cropped images is negligible. 2221
randomly selected images are annotated by human experts into
8 classes, as shown in Fig. 2(a), where these are used to validate
the performance of the proposed method. Fig. 2(b) shows thehorizontal distribution of each ground truth class in the dataset.
The figure shows that the classes form continuous spatial patterns along the sparse survey trajectories. Fig. 2(c) shows thedepth distribution of annotated images in each class together theclass labels. The figure shows that Kelp (A) is found at shallow
depth ranges where energy from the sun can reach. High Relief
Coral (B) and Low Relief Reef (C) start to appear at the depth of
40 m and 45 m, respectively. Other classes (D) - (H) also exhibit
unique depth distributions, though there is considerable overlap
beyond 50 m depth.
Horizontal location and depth estimates for each image are
generated based on the Simultaneous Localisation and Mapping(SLAM) pipeline described in [25]. Georeference errors smaller
than σloc in (3) or σdep in (7) do not affect the optimisation.
Where SLAM or other global localisation methods such as
ultra-short baseline or long-baseline acoustic positioning are
not used, horizontal position errors accumulate at a rate of
approximately 1% distance travelled using typical AUV navigational sensor suites [26]. In practical terms, this means thatthe position uncertainty between sequentially taken images will
be negligible. For images taken nearby but with a longer periodof separation, the position uncertainty should be estimated using
established methods (e.g. an extended Kalman filter) and where
the uncertainty exceeds σloc , the pair should be rejected. Error
accumulation does not occur when using commercial gradepressure and altitude sensors to determine seafloor image depth

To investigate the effectiveness of the proposed regularisation,
the autoencoder is trained (i) without regularisation, (ii) with
Lloc , (iii) with Ldep , (iv) with both Lloc and Ldep on all 32 097
images in the dataset. AlexNet [7] with batch normalisation isused as the encoder architecture, and its inverse is used as the
decoder where the number of dimensions of the encoder output(equal to the number of dimensions of the decoder input) is set to
16 in accordance with our previous work [19]. The autoencoder
weights are initialised with the values of AlexNet pre-trainedon ImageNet. A mini-batch size of n∗ = 256 is applied andrandom rotation, shifting, flipping and colour distortions areapplied for data augmentation. In the experiments where either Lloc or Ldep are applied, n∗ /2 images are selected fromthe metadata space neighbourhoods of each randomly selected
sample, and remaining n∗ /2 images are selected randomly fromthe entire dataset. σloc in (3) is set to 10.0 m, and σdep in (7)is set to 1.0 m since image appearance is expected to show
some degree of correlation with horizontal location and depth
within these ranges. Preliminary experiments indicated that themethod is not highly sensitive to these parameters, where σloc
values ranging from 3.0 to 20 m only had a marginal impact on
performance. This is favourable for practical application sinceextensive parameter tuning via trial and error is not necessary.
Both λloc and λdep in (2) are set to 1 × 105 , and a learning
rate of lr=1 × 10−5 is used for the Adam optimiser. Thesehyperparameters are experimentally determined so that all loss
terms that are applied decrease during training. This is alsofavourable in practical terms since decrease of the loss function
is a necessary condition for successful training, where mostworkflows already confirm this happens before proceeding withfurther analysis. The number of epochs is set to 100 and eachexperiment configuration is executed three times.
C. Evaluation Metrics
The representation learning performance is evaluated basedon the classification accuracy achieved using the acquired representations. The classifiers used to assess performance consistof a k-Nearest Neighbour with k=1 (1-NN), a Gaussian Processclassifier (GP), Random Forest (RF), Support Vector Machinewith Linear kernel (L-SVM) and with Radial basis function
kernel (R-SVM). A 10-fold cross validation is performed to
examine each autoencoder, where three autoencoders are usedin each training configuration. To reduce the effect of classimbalance, the cost functions of RF, L-SVM and R-SVM arebalanced considering the class counts. The F1 score (macro
average) is used for performance evaluation, where we consider
all class to be of equal importance. Though this experimentconsiders classification to evaluate accuracy, the higher scoreindicates that the obtained representations are effective at describing the images, and so form a favourable basis for otherapplications such as clustering, contents retrieval, and use inobservation-aware path planning methods.

Authorized licensed use limited to: University of Tokyo. Downloaded on April 11,2022 at 10:16:28 UTC from IEEE Xplore. Restrictions apply.

7820

IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 6, NO. 4, OCTOBER 2021

Fig. 2. Overview of the Tasmania dataset. The number of expert human annotations in each image class are shown together with example images in a), with classlabels: A - Kelp, B - High Relief Reef, C - Low Relief Reef, D - Patch Reef, E - Reef & Sand, F - Screw Shell Rubble, G - Screw Shell Rubble & Sand, and H Coarse Sand. The horizontal spatial distribution of the human annotated classes are shown in b) and the depth distribution of each class is shown in c), where thesame colour scheme has been used throughout the figure. The horizontal location loss Lloc regularises learning based on the horizontal distribution of the images,
and the proposed depth loss Ldep regularises learning based on their depth distribution.

TABLE II
F1 MACRO AVERAGE SCORES FOR EACH REGULARISATIONCONFIGURATION AND CLASSIFIER

The convolutional autoencoder is trained (i) without regularisation, (ii) with Lloc , (iii)with Ldep , (iv) with Lloc and Ldep . Five different classifiers are trained on the autoencoder embedded representations (1-Nearest Neighbour, Random Forest, Gaussian Processclassifier, Linear kernel Support Vector Machine (SVM) and Radial basis function SVM.
The F1 Macro Average is computed based on human labels.

D. Result
Table II shows the mean and standard deviation of the F1scores for each autoencoder training configuration and classifier.
For four of five classifiers; 1-NN, GP, L-SVM and R-SVM, theautoencoders trained with both Lloc and Ldep (configuration

Fig. 3. Per-class F1 -scores and their macro average for (i) no regularisation,(ii) horizontal location regularisation, (iii) depth regularisation and (iv) horizontal location and depth regularisation. R-SVM is used as the classifier in thisplot.

(iv)) show the best performance among the four configurations. For RF, configuration (ii), where only Lloc is applied,has the best score. However, the difference between (ii) andAuthorized licensed use limited to: University of Tokyo. Downloaded on April 11,2022 at 10:16:28 UTC from IEEE Xplore. Restrictions apply.

YAMADA et al.: LEVERAGING METADATA IN REPRESENTATION LEARNING WITH GEOREFERENCED SEAFLOOR IMAGERY

7821

Fig. 4. The t-SNE latent representation learnt by the proposed method for both horizontal location and depth regularisation, i.e. configuration (iv). Representative
images are selected based on k-means clustering for (a) and hierarchical k-means clustering for (b). The colours represent the classes determined by R-SVM andare used for illustrative purposes only.

(iv) is marginal. Configurations (ii) - (iv) perform better thanconfiguration (i), where no regularisation is applied, for allclassifiers, achieving an average performance gain of (ii) 9.4 %,(iii) 6.9 % and (iv) 10.9 %, respectively. The results show thathorizontal location metadata is more effective for learning latent
representations than depth for this dataset. However, using bothof horizontal location and depth information generally improves
performances, and never causes any significant degradation. The
biggest gains in performance are seen for the R-SVM classifier,where an improvement of (ii) 12.5%, (iii) 8.7% and (iv) 15.1%,
are seen respectively compared to no regularisation (i). Anothernoticeable point is that for L-SVM, configuration (iii) showsa better score better than (ii). Among the five classifiers usedin the experiment, L-SVM is the only linear classifier, whichmakes it relatively robust against over-fitting. A different trend
is observed compared to the other classifiers with depth only
regularisation performing favourably. A possible explanation
for this is that some over-fitting may be taking place with the
non-linear classifiers when only depth regularisation is used.
Fig. 3 shows the per-class F1 scores of the best performingclassifier (R-SVM) for regularisation configurations (i)–(iv).
Configurations (ii)–(iv) are superior to configuration (i) for allclasses. Horizontal location regularisation (ii) performs better
than depth regularisation (iii) for all classes except for C. The
relative performance improvement with metadata regularisation
is most significant for classes D, E, and H (24.9%, 33.5%,
and 52.6% between (i) and (iv), respectively), which have relatively small populations in the dataset. This can be explained
as optimising only the autoencoder reconstruction loss Lrec
potentially leads to focusing on the appearances of majority
classes, where the proposed regularisation avoids this form ofover-fitting by effectively prioritising patterns in classes withsmaller populations.

An important characteristic of the proposed method is thatboth regularisation methods can be applied without risk ofsignificant performance degradation. This is due to the use oft-distributions and the loose regularisation constraints imposed
during the loss function optimisation based on probabilistic
distributions. We see this characteristic where configuration (iv)
leads to and overall improvement in performance, and betterclass scores than configurations (ii) and (iii) for most classes.
Where the scores for classes C, F and G are slightly degraded,
the difference is negligible. Although horizontal location regularisation is generally more effective than depth regularisation
for this dataset, the ability to improve performance using only
depth information is valuable as accurate horizontal localisationin GPS denied subsea environments requires expensive navigational sensors that may not be available on some low cost AUVs
and Remotely Operated Vehicles (ROVs). On the other hand,
depth sensors are relatively cheap and so are available on almost
all underwater platforms.
E. Application to Seafloor Survey
Latent representations can be applied to efficiently understand the characteristics of a dataset. One way to do this is byautomatically identifying images that are most representativeof the variety of scenes that exist in the data. Fig. 4 shows
the automatically selected representative images, overlaid on
the representations of Tasmania dataset using a t-SNE visualisation [21]. In Fig. 4(a), k-means clustering is applied tothe acquired latent representations, and the images closest toeach of the k centroids are selected as representative images.
Here, we use k=8 which is automatically determined basedon the elbow-method [27]. In Fig. 4(b), Hierarchical k-meansclustering [28] is applied to identify a further k  =3 within each

Authorized licensed use limited to: University of Tokyo. Downloaded on April 11,2022 at 10:16:28 UTC from IEEE Xplore. Restrictions apply.

7822

IEEE ROBOTICS AND AUTOMATION LETTERS, VOL. 6, NO. 4, OCTOBER 2021

original cluster. This allows for representation of the range andsequential transitions of seafloor scenes. The results show that a
relatively small number of representative images automatically
identified by the system can efficiently describe the variety ofscenes found in a dataset consisting of more than 30 k images,
including representative examples of classes with a small population. This is valuable for remote transmission of exemplary
data over the limited bandwidths available using long-range underwater acoustics communications, or global communication
satellites when platforms are at the water surface. Representative
images may also benefit low-shot training of supervised andsemi-supervised classifiers.
V. CONCLUSION
We have proposed a novel autoencoder regularisation methodthat can leverage any number and combination of vector orscalar metadata for seafloor image representation learning. The
regularisation is effective when two images that are close in
their metadata space tend to be more similar in appearance.
By optimising loss functions using the KL divergence andt-distributions, it is possible to mitigate over-regularisation by
metadata and avoid significant performance degradation when
multiple sources of metadata are applied to regularise learning.
The self regulating latent representation learning method wasapplied to a dataset consisting of more than 30 k images taken
during 6 AUV dives. Validation against 2.2 k expert humanannotations shows that:r Combining multiple sources of metadata regularisation
can outperform single metadata regularisation using the
proposed method. Regularising learning using depth andhorizontal location metadata improves the performance offive classifiers operating on the latent representations by
an average of 10.9% compared to a standard convolutional
autoencoder, with the R-SVM classifier showing the largest
gain in performance at 15.1%.
r Horizontal location regularisation is more effective thandepth regularisation for the sparse transect dataset analysedin this work, achieving an average improvement of 9.4%
(as opposed to 6.9%) across five classifiers, and 12.5%(as opposed to 8.7%) for the best performing classifier.
However, combining both in metadata regularisation reliably outperforms individual regularisation and never significantly degrades performance.
r The acquired latent representations allow representative
images of large datasets with imbalanced class distributions to be automatically identified in a fully unsupervisedway, which can help achieve an efficient understanding ofunderwater scenes and be applied to adaptive path planning
using visual information.
REFERENCES[1] A. Bender, S. B. Williams, and O. Pizarro, “Autonomous explorationof large-scale benthic environments,” in Proc. IEEE Int. Conf. Robot.
Automat., 2013, pp. 390–396.
[2] J. Shields, O. Pizarro, and S. B. Williams, “Feature space exploration forplanning initial benthic AUV surveys,” 2021, arXiv:2105.11598.

[3] G. Flaspohler, V. Preston, A. P. Michel, Y. Girdhar, and N. Roy,
“Information-guided robotic maximum seek-and-sample in partially observable continuous environments,” IEEE Robot. Automat. Lett., vol. 4,
no. 4, pp. 3782–3789, Oct. 2019.
[4] Y. Girdhar and G. Dudek, “Exploring underwater environments withcuriosity,” in Proc. Can. Conf. Comput. Robot Vis., 2014, pp. 104–110.
[5] M. Bewley, N. Nourani-Vatani, D. Rao, B. Douillard, O. Pizarro, and S.
B. Williams, “Hierarchical classification in AUV imagery,” in Field Serv.
Robot. Cham: Springer, 2015, pp. 3–16.
[6] D. Rao, M. De Deuge, N. Nourani-Vatani, S. B. Williams, and O. Pizarro,“Multimodal learning and inference from visual and remotely sensed data,”
Int. J. Robot. Res., vol. 36, no. 1, pp. 24–43, 2017.
[7] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097–1105.
[8] D. Steinberg, A. Friedman, O. Pizarro, and S. B. Williams, “A bayesiannonparametric approach to clustering data from underwater robotic surveys,” in Proc. Int. Symp. Robot. Res., vol. 28, 2011, pp. 1–16.
[9] O. Beijbom, P. J. Edmunds, D. I. Kline, B. G. Mitchell, and D. Kriegman,“Automated annotation of coral reef survey images,” in Proc. IEEE Conf.
Comput. Vis. Pattern Recognit., 2012, pp. 1170–1177.
[10] J. W. Kaeli and H. Singh, “Online data summaries for semantic mapping
and anomaly detection with autonomous underwater vehicles,” in Proc.
IEEE OCEANS Genova, 2015, pp. 1–7.
[11] U. Neettiyath et al., “Deep-sea robotic survey and data processing methods
for regional-scale estimation of manganese crust distribution,” IEEE J.
Ocean. Eng., vol. 46, no. 1, pp. 102–114, Jan. 2021.
[12] T. Ojala, M. Pietikäinen, and T. Mäenpää, “Multiresolution gray-scale androtation invariant texture classification with local binary patterns,” IEEE
Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971–987, Jul. 2002.
[13] J. Yang, K. Yu, Y. Gong, and T. Huang, “Linear spatial pyramid matching
using sparse coding for image classification,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit., 2009, pp. 1794–1801.
[14] A. Mahmood et al., “Deep image representations for coral image classification,” IEEE J. Ocean. Eng., vol. 44, no. 1, pp. 121–131, Jan. 2019.
[15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016,
pp. 770–778.
[16] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation ofword representations in vector space,” 2013, arXiv:1301.3781.
[17] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors forword representation,” in Proc. Conf. Empirical Methods Natural Lang.
Process., 2014, pp. 1532–1543.
[18] N. Jean, S. Wang, A. Samar, G. Azzari, D. Lobell, and S. Ermon, “Tile2vec:
Unsupervised representation learning for spatially distributed data,” inProc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, pp. 3967–3974.
[19] T. Yamada, A. Prügel-Bennett, and B. Thornton, “Learning features fromgeoreferenced seafloor imagery with location guided autoencoders,” J.
Field Robot., vol. 38, no. 1, pp. 52–67, 2021.
[20] M. Modasshir and I. Rekleitis, “Enhancing coral reef monitoring utilizing
a deep semi-supervised learning approach,” in Proc. IEEE Int. Conf. Robot.
Automat., 2020, pp. 1874–1880.
[21] L. v. d. Maaten and G. Hinton, “Visualizing data using t-SNE,” J. Mach.
Learn. Res., vol. 9, pp. 2579–2605, 2008.
[22] L. Jing and Y. Tian, “Self-supervised visual feature learning with deepneural networks: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., 2020,
doi: 10.1109/TPAMI.2020.2992393.
[23] S. B. Williams et al., “Monitoring of benthic reference sites: Using anautonomous underwater vehicle,” IEEE Robot. Automat. Mag., vol. 19,
no. 1, pp. 73–84, Mar. 2012.
[24] T. Moltmann et al., “A global ocean observing system (GOOS), delivered
through enhanced collaboration across regions, communities, and newtechnologies,” Front. Mar. Sci., vol. 6, p. 291, 2019.
[25] I. Mahon, S. B. Williams, O. Pizarro, and M. Johnson-Roberson, “Efficient view-based SLAM using visual loop closures,” IEEE Trans. Robot.,vol. 24, no. 5, pp. 1002–1014, Oct. 2008.
[26] L. Paull, S. Saeedi, M. Seto, and H. Li, “AUV navigation and localization:
A review,” IEEE J. Ocean. Eng., vol. 39, no. 1, pp. 131–149, Jan. 2014.
[27] V. Satopaa, J. Albrecht, D. Irwin, and B. Raghavan, “Finding a” kneedle”
in a haystack: Detecting knee points in system behavior,” in Proc. 31st Int.
Conf. Distrib. Comput. Syst. Workshops, 2011, pp. 166–171.
[28] D. Nister and H. Stewenius, “Scalable recognition with a vocabulary tree,”in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2,
2006, pp. 2161–2168.

Authorized licensed use limited to: University of Tokyo. Downloaded on April 11,2022 at 10:16:28 UTC from IEEE Xplore. Restrictions apply.