Semi-Supervised Audio Representation Learning for Modeling Beehive Strengths
Authors: Tony Zhang
Szymon Zmyslony
Sergei Nozdrenkov
Matthew Smith
Brandon Kingsley Hopkins
Venue:  (2021)
Keywords: bee
beehive
audio
sound
computational ethology
deep learning
representation learning
semi-supervised learning
modeling
population
disease

Abstract: Honey bees are critical to our ecosystem and food security as a pollinator, contributing 35% of our global agriculture yield. In spite of their importance, beekeeping is exclusively dependent on human labor and experience-derived heuristics, while requiring frequent human checkups to ensure the colony is healthy, which can disrupt the colony. Increasingly, pollinator populations are declining due to threats from climate change, pests, environmental toxicity, making their management even more critical than ever before in order to ensure sustained global food security. To start addressing this pressing challenge, we developed an integrated hardware sensing system for beehive monitoring through audio and environment measurements, and a hierarchical semi-supervised deep learning model, composed of an audio modeling module and a predictor, to model the strength of beehives. The model is trained jointly on audio reconstruction and prediction losses based on human inspections, in order to model both low-level audio features and circadian temporal dynamics. We show that this model performs well despite limited labels, and can learn an audio embedding that is useful for characterizing different sound profiles of beehives. This is the first instance to our knowledge of applying audio-based deep learning to model beehives and population size in an observational setting across a large number of hives.

Under review as a conference paper at ICLR 2021
SEMI-SUPERVISED AUDIO REPRESENTATION LEARN- ING FOR MODELING BEEHIVE STRENGTHS
Anonymous authors
Paper under double-blind review
ABSTRACT
Honey bees critical ecosystem and food security as a pollinator, con- tributing 35% of our global agriculture yield (Klein et al. (007). In spite of their importance, beekeeping is exclusively dependent on human labor and experience- derived heuristics, while requiring frequent human checkups to ensure the colony is healthy, which can disrupt the colony. Increasingly, pollinator populations are declining due to threats from climate change, pests, environmental toxicity. mak- ing their management even more critical than ever before in order to ensure sus- tained global food security. To start addressing this pressing challenge, we de- veloped an integrated hardware sensing system for beehive monitoring through audio and environment measurements, and hierarchical learning model, composed of an audio modeling module and a predictor, to model the strength of beehives. The model is trained jointly on audio reconstruction and prediction losses based on human inspections, in order to model both low-level audio features and circadian temporal dynamics. We show that this model per- forms well despite limited labels, and can learn an audio embedding that is useful for characterizing different sound profiles of beehives. This is the first instance to our knowledge of applying audio-based deep learning to model bechives and population size an observational setting across large number of hives.
supervsde deep
INTRODUCTION
Pollinators are one of the most fundamental parts of crop production worldwide (Klein al. 2007 Without honey bee pollinators, there would be substantial decrease in both the diversity and yield of our crops, which includes most common produce (van der Sluijs Vaage) (016). As model organism, bees are also often studied through controlled behavioral experiments, as they exhibit complex responses to many environmental factors, many which are yet to be fully understood. A colony of bees coordinate its efforts to maintain the overall health, with different types of bees tasked for various purposes. One of the signature modality of characterizing bee behavior through the buzzing frequencies emitted through the vibration of the wings, which can correlate with various properties of the surroundings, including temperature, potentially allowing for a descriptive 'image' of the hive in terms of strength (Howard et al. 2013 Ruttner 1988).
However, despite what is known about honey bees behavior and their importance in agriculture and natural diversity, there remains a substantial gap between controlled academic studies and the field practices carried out (López-Uribe & Simone-Finstren long-tenured experience to derive heuristics for maintaining colonies, which necessitates frequent visual inspections of each frame of every box, many of which making up single hive. During each inspection, beekeepers visually examine each frame and note any deformities, changes in colony size, amount of stored food, and amount of brood maintained by the bees. This process is labor in- tensive, limiting the number of hives that can be managed effectively, As growing risk factors make human inspection more difficult at scale, computational methods are needed in tracking changing hive dynamics on a faster timescale and allowing for scalable management. With modern sensing hardware that can record data for months and scalable modeling with state-of-the arr tools in ma- chine learning, we can potentially start tackling some challenges facing the management of our pollinators, key player in ensuring food security for the future.
2019). In particular, beekeepers use their

---

Under review as a conference paper at ICLR 2021
2 BACKGROUND AND RELATED WORKS
Our work falls broadly in applied machine learning within computational ethology, where automated data collection methods and machine learning models are developed to monitor and characterize biological species in natural or controlled settings (Anderson & Perona 2014). In the context of honey bees, while there has been substantial work characterizing bee behavior through controlled audio, image, and video data collection with classical signal processing methods, there has not been a large-scale effort studying how current techniques in deep learning can be applied at scale to the monito-moiioriing of beehives in the field.
Part of the challenge lies in data collection. Visual-sensing within beehives is nearly impossible given the current design of boxes used to house bees. These boxes are heavily confined with narrow spaces between many stacked frames for bees to hatch, rear brood, and store food. This makes it difficult to position cameras to capture complete data, without a redesign of existing boxes. Envi- ronment sensors, however, can capture information localized and humidity. Sound, likewise, can travel across many stacked boxes, which are typically made from wood and have good acoustics. Previous works have explored the possibility of characterizing colony status with audio in highly stereotyped events, such as extremely diseased vs healthy bee- hives KRobles-Guerrees
larger region, such as temperature
et al., 2017) or swarming (Krzywoszyja al. 2018 Ramsey et al. 2020), where the old Queen leaves with large portion of the original colony. However, we have not seen work that attempt to characterize more sensitive measurements, such as population of beehives, based on audio. We were inspired by these works and the latest advances in hardware sensing and deep learning audio models to collect audio data in a longitudinal setting across many months for large number of managed hives, and attempt characterize some of the standard hive inspection items through machine learning.
While audio makes it possible to capture more complete picture of the inside of hive, there are still challenges related to data semantics in the context of annotations. Image and video data can be readily processed and labeled post-collection if the objects of interest recognizable. However, with honey bees, the sound properties captured by microphones are extremely difficult to discrimi- nate, even experts, due to the fact that the sound is not semantically meaningful, and microphone sensitivity deviations across sensors makes it difficult to compare data across different hives. Thus, is not possible to retrospectively assign labels to data, making humans inspections during data collection the only source of annotations. As beekeepers cannot inspect a hive frequently due to the large number of hives managed and the potential disturbance caused to the hive, the task becomes few-shot learning.
In low-show learning for audio, various works have highlighted the usefulness of using semi- supervised or unsupervised objectives and/or learning an embedding of audio data, mostly for the purpose of sound classification or speech recognition (Jansen et al.] 2020 Lu et al.) 2019). These models typically capture semantic differences between different sound sources. We were inspired by the audio classification work with semi-supervised or contrastive-leamiigg objectives to build an ar- chitecture that could model our audio and learn embedding without relying only on task- specific supervision. Unlike previous audio datasets used in prior works, longitudinal data is unlikely to discretize into distinct groups due the slower continuously shifting dynamics across time on the course of weeks. Therefore, we make the assumption that unlike current audio datasets which con- tain audio from distinct classes that can be clustered into sub-types, our data more likely occupy smooth latent space. due to the slow progression in time of changing properties, such as the transition between healthy and low-severity disease, or changes in the size of the population, as bee colonies increase by only around one frame per week during periods of colony growth (Russell et al. 2013 Sakagami & Fukuda 1968).
METHODS
Hive Setup Each hive composes of multiple 10-frame standard Langstroth boxes stacked on top of one another, with the internal sensor located in the center frame of the bottom-movt most box, and the external sensor on the outside side wall of the box. This sensor placement is based prior knowledge that bees tend to collect near the bottom box first prior to moving up the tower (Winston) 1987.. Due to difficulties in obtaining data that would span the spectrum of different colony sizes
2

---

Under review as a conference paper at ICLR 2021
without intervention in timely manner, up hives of varying sizes in the beginning to capture a range of populations. This allowed our dataset to span a number of frame sizes, from to 23 for bee frames, and 0 to 11 for brood frames. Aside from these manipulations, all other observed effects, such as progression of disease states, are of natural causes free from human intervention.
3.1
DATA COLLECTION
6142
4096
45
0.75
0.50
Figure 1: Collected multi-modal data from our custom hardware sensor for one hive across one month. The sensor records minute-long audio sample and point estimates of the temperature. humidity, and air pressure for both inside and outside the hive every 15 minutes during all hours of day. Black line indicates measurements recorded internal of bee box; gray line indicates external of box. Internal measurements are consistent for temperature and humidity as bees thermoregulate internal hive conditions (Stabentheiner et al. 2010).
Sensor Data Given prior works that showed the possibility of characterizing honey bee colonies through audio, we developed nattery-powered sensorbars that can be fitted to single frame of standard Langstroth bee box. Each sensor is designed for longitudinal data collection over the span of many weeks on single charge. The sensor records sub-sampled data every 15 minutes, at all hours of day. Each multi-modal data sample composes a one-minute audio sample and point estimates of the temperature, humidity, and pressure, for both inside and outside the box (Fig. []] For the purpose of the daily-snapshot model described in this work, we use data from all days with 96 samples collected. In sum, have collected 1000 total days of data, across 26 hives, with up to 180 corresponding human inspection entries. These inspection entries captured information related hive status, which for our purpose are frames of bees, frames of brood, disease status, and disease severity.
Inspections We used data from one inspector for all collected data used in this work in order to increase annotation consistency. The inspector performed observation in each hive roughly once per week, during which they visually examine each individual frames in all boxes for that hive. The hives placed 2 meters apart from one another such that cross contamination of audio is unlikely, given the sensor is isolated to within each stack boxes. For frame labels, the inspector visually examine each frame to determine if that frame is least 60% covered, given which it would be added the total frame count. We prevent overcrowding on each frame by introducing empty frames whenever necessary, such that each frame covered most up 90%, is common practice. This allows us to obtain a lowerbound on the error range of our inspections at around +200%. During the same inspection, the inspector also check for the presence of any diseases and its severity. Severity scores between none, low, moderate, and severe, where low corresponds to single observation of diseased bees, moderate for several observations of disease, and severe for prevalent signs of discase.
NETWORK
Given the difficulty of collecting ground truths due to the nature of our data, we sought out to develop remi-supervised model and leverage large number of audio samples. Additionally, behavior from bees leaving and returning to bechives means that data from one full-day circadian cycle must be used for predictions in order model same-day variations. Therefore, developed model
3

---

Under review as a conference paper at ICLR 2021
trained on hierarchical objectives allow for modeling both low-level audio features minute- long basis, as well any complex temporal dynamics within a given day. We do not consider longer time horizon for this work the focus is in modeling snapshot of the hive's current state, not where will be in the future. Given prior works characterizing beehive sound profiles in lab settings, we know that local audio features critical, audio strength along certain known frequencies correlate with different behaviors and types of bees, which could potentially allow for discerning population sizes and disease statuses.
4.1 ARCHITECTURE
Embediing Module
Predicion (Enooded)
Predictor
Disess Severity
D sease Type
Predictoon (Predidor)
8L-05
SL-M4
81-1
81
One Day
Figure 2: Hierarchical Generative Prredicciiniiiiiinnnn Network. s: point estimates of environmental factors (temperature, humidity, air pressure), latent variables. The encoder has 4 convolutional layers layers. The Predictor
with max- max-pooling in between, and the decoder has cunsposed-convolutionn receives concatenated inputs from the encoder's latent variables well sensor data for 96 samples across one day. The predictor attaches to multiple heads, sharing parameters to encourage shared- representation learning and regularize based multi-task objective.
The model composes of two components in hierarchy and purpose: an audio embedding module, and temporal prediction module (Fig. 2). The embedding module learns a very low-dimensional representation of each one-minute long audio samples, while sharing its parameters across 96 samples across each day. Each encoder-decoder is convolutional variational autoencoder. This embedding module outputs a concatenated audio latent space, which is 96 x dz. representing all samples from the beehive across day. The embedding module is trained via variational infer- ence on maximizing the log likelihood of the reconstruction, which is 56 x 56 downsampled mel spectrogram, same the input. The embedding module is pre-trained optimize each sample sep- arately, and not capture temporal dynamics explicitly. It is used to learn feature filters that are less dependent on the prediction loss downstream, which can bias the model due limited data that has assigned inspection entries.
The predictor is shallow feed-forward network, designed to prevent overfitting and model simple daily temporal dynamics. It is trained only on data with matching inspection labels. The predictor takes in all concatenated latent variables from 96 audio samples of each day, along with correspond- ing 96 samples of environmental data, which includes temperature, humidity, and pressure for inside and outside the box. The sensor input is less important for predicting the momentary population and disease status than to normalize the observed audio, given that activity is known vary with respect to temperature and humidity. The predictor is then connected to multiple heads, jointly-traine multi-task objectives. The parameter sharing used in the predictor also designed to reduce overfit- ting and to capture similar representations, as the predicted tasks are likely similar in nature (disease and population).
on

---

Under review as a conference paper at ICLR 2021
Objectives The embedding module is trained jointly audio sample reconstruction via the idence lowerbound (Eq. D as well as global prediction loss across a given day, backpropagated through the latent variables. The generator is pre-trained for ~8000 iterations in order to learn stable latent representations before prediction gradients are propagated. The predictor is trained via multi-task prediction losses. This training then continues until all losses have converged and sta- bilized. The multi-task objective is composed of Huber loss (Eq. ( for frames and disease severity regressions and categorical cross-entropy for disease classification.
logp(z) 2 ((x) Bz>a(=)>) logp(x|2) DxLlg(<l=))ln(e]]
I(y.f(x) { 8 (1v- 1ly-f(ee f(x)) 6/2) otherwise.
for -y- <
(2)
4.2 EVALUATION
Due to the nature of our annotation collection, deviations can be expected our ground truths for both diseases and frames. In particular, inspections often occur on days with incomplete data samples. In order to reduce uncertainty around our inspection entries, we curated our training and validation sets by excluding inspections that were recorded more than two days away from the nearest day with a complete set of sensor data, due to hardware or battery issues causing some days to record incomplete data. This led us a inspection-pairee dataset of 38 samples across 26 hives, spanning 79 days. Recognizing the limited sample size, we carry out 10-fold validation with all models evaluated in our model comparisons. In addition, reduce the possibility of cross contamination between the training and test set due to sensor similarities, we do not train and validate on the same sensor hive, even if the datapoints are months apart. This is done despite our sensors not being cross-calibe crossccalbbraed, as we wanted to see whether our predictions are able to fully generalize across completely different hives without the need for fine-tuned sensors, which can costly to implement.
To account for the variance around ground truths of frames, we compute cumulative density func- tions for percentage differences between all frame predictions and inspections, in order to exam- ine the fraction predictions that fall within our ground truth error range's lowerbound, which is +20% of the assigned label (Fig. 4). We compute validation scores for each partitioned validation set, averaged across all 10 groups and for each training iteration, in order to gather a more complete understanding of how each model's training landscape evolves, and also to assess how easily each model overfits (Fig. Dh. For all evaluation results, we show mean losses scores across 10 separate runs, each composed of training model 10 times on 10-fold validation scheme.
RESULTS
5.1 MODEL COMPARISONS
We compared several versions of the model with ablation in order to understand the effect of each module or objective on prediction performance (Table [ First, we compared GPN without the sound-embedding module, and trained it on environment features and hand-crafted audio features, which is the norm prior literature. These handcrafted audio features include fast Fourier trans- formed (fft) audio and mean audio amplitude, where fft is sampled across 15 bins between 0 to 2667 Hz, which is just above what prior literature have shown bee related frequencies to be under (Howard et al. 2013 Qandour et al. [0144) We also trained version of GPN purely on audio with- out any environment data, which we expect is needed to remove the confounding effects of external environment on bee behavior.
MLP trained on sensor data and fft features. We found that the fully supervised MLP trained on handerafted features worked well, given a carefully selected bin size, performing slightly worse than GPN trained on spectrograms (Fig, B). We think this is evidence that the GPN is learning filters across the frequency domain rather than the temporal domain, frequencies likely matter more than temporal structure within each minute-long sample given the sound lack any fast temporal variations.
5

---

Under review as a conference paper at ICLR 2021
19
01
A
H
I
8
Ho
HO
HP
18
d
18
d1
HP
Ho
d
al
1d
10
10
04
00
M
HD
12
00
HO
18
8
8
0
10
30
25
46
Sample
Figure 3: Model predictions on combined 10-fold validation set, where each fold validates on 1-3 hives that the model was not trained on. Ranges / crosses indicate inspection, circles indicates pre- dictions. We indicate frame inspection label with a range of 20%% which represents an approximate lowerbound on the eπror range of the ground truth. These samples comes from 26 hives sampled across the span of 79 days.
Table Task prediction performance. Env: environment data. fft fast Fourier transformed features. labeled: data with assigned inspections. Unlabeled: all data samples. Metric for frames and disease severity: Huber loss; disease type: accuracy.
Task
MLP Env, fft
GPN Labeled
GPN Unlabeled
Frames
0,0206 = 2.7e-3
0.227 + 4.2e-1
0.0166 + 2.1e-3
Disease Type
0.757 + 1.6e-2
0.689 + lc-1
0.781 = 1.20-2
Disease Severity
0.0343 = 6.7c-3
0.259 + 5.1e-1
0.0331 + 4.70-3
GPN trained on all vs only labeled data. As we collect 96 audio samples per day across many months, many of these datapoints are not associated with matching annotations for task predictions. However, given that our model can be trained purely training on
generative objective, we compared models our data on days with complete 96 samples, vs only on the data with assigned inspection entries. By including unlabeled data, are able increase valid dataset from 38 to 856. We found that this model performed better for all tasks, likely attributable to learning more generalizable kernels in the encoders (Fig.
5.2 EFFECTS OF ENVIRONMENT MODALITIES PERFORMANCE
As this is the first work we have seen using combined non-visual multi-modal data for predicting beehive populations, we wanted to understand which modalities are most important prediction performance. Thus, we tested the effects of sensor modality exclusion performance. We trained multiple GPNs, removing each environmental modality from the sensor data used for predictor training. and observe the increases in validation loss decrease in accuracy (Table 2}. We found that when compared to GPN trained on all sensor modalities Table ! removing temperature had the greatest effect performance, while humidity and pressure had less effect. This is possibly because thermoregulation is an important behavioral trait that is highly correlated with a healthy hive. Since air pressure unlikely capture this property, humidity and temperature most important, with temperature being sufficient. We do believe that these modalities are important for prediction if hives are placed radically different geographic locations with varying altitudes and seasonal effects, which is beyond the scope of our dataset.
likely the
5.3 LEARNED EMBEDDING AND AUDIO PROFILES
Due to the nature of the purely observational data resulting in limited samples of diseased states, examined the generative model's embeddings and outputs from the decoder evaluate the gen- erative model's s ability to model diseased states. In particular, we wanted to understand if it has learned to model variations in audio spectrograms, and if purely unsupervised model trained

---

Under review as a conference paper at ICLR 2021
Brood
1.0
Cummuisvv Cum.istveProbaai Probaklty
MLP
0.6
GPN
[Lowerbouns]
0.0
0%
25%
50%
75%
2000
0%
25%
50%
75%
100%
2.5
7.5
10.0
0.0
2.5
5.0
7.5
10.0
of 1961
(%)
Figure 4: Frame prediction performance comparing supervised MLP on fft audio features and semi- supervised GPN. CDFs of absolute differences and percentage difference shown.
Table 2: Environment modality exclusion's effect on performance. Mean shown with standard deviation computed for 10 runs of each group, validated 10-fold.
Task
Humidity
Temperature
Pressure
Frames
0.0189 = I.6c-3
0.143 + 4.3c-1
0.0207 4c-3
Disease Type
0.763 + 4e-9
0.724 + 1.4e-1
0.763 + le-2
Disease Severity
0.0366 + 9e-2
0.220 + 6.7e-1
0.0315 + 4.9e-3
spectrograms alone or with prediction signal would show inherent differences between diseased and healthy states.
We sampled audio spectrograms from the test set and examined how well our embedding model has learned to reconstruct the spectrograms, and in general, we found that the embedding model has learned to model the frequency bands we see in our data (Fig. [ as well as the magnitude of each band. We also sampled from latent variables and examined the differences in learned sound profiles extracted from the decoder, and found that the two dimensions correspond to differences across the frequency spectrum (Fig. a)..
We also projected audio data from the dataset into the embedding space across each day, representing 96 audio samples latent dimensions. For visualization, we then compute PCAs of each full-day latent representation. The first two PCA components captured significant portion of the variance of the embedding space, with PC-1 and PC-2 representing respective 74.17% and 10.61% of the variance. We color-mapped this PCA space with disease severity: healthy, low disease, medium disease, and high disease, with the goal of observing how these diseased states are organized in the latent space. We found that within each disease class there was relatively consistent organiza- tion. The audio samples that are classified as healthy occupied a more distinct region within this space, and the samples classified diseased separated out primarily along PC-1. The low disease severity class was the most discriminated within the space, followed by the medium disease sever- ity, then high disease severity. We were interested in what features of the frequency spectra were learned within the embedding model, thus extracted spectrograms across 24 hour window for each projected point and computed the mean across each disease severity class. We compared these fre- quency spectra to the corresponding latent variables. Within the healthy audio samples, we observed changes in magnitude and broadening of spectral peaks across the day, which could be associated
Figure Sampled test spectrograms (top) and reconstructions (bottom).

---

Under review as a conference paper at ICLR 2021
PC1. 2612%
Figure 6: left: Sampled latent space decoding. The jointly trained generative model audio data allows us sample the space occupied by the two latent variables. Distinct audio profiles show the model captured range of beehive sound profiles. center: Visualization of audio samples in latent space across one day reduced to two PCA dimensions, color-mapped to disease severity. right: The average frequency spectra for each disease class synchronized across 1-hour window.
with honey bee circadian rhythms. and similar changes are observed in the corresponding latent variables (Eban-Rowle Rothschild & Bloch 2008 Ramsey et al. 2018). The low disease severity class, which shows the most separation in PCA space, has the most distinet spectrogram signatures. In particular, within the low-disease severity magnitude that was consistent across 24 hours. As the disease class progressed from low to medium and high severity, we observed reduction in magnitude and any circadian changes become less ap- parent. These characteristics appear to be consistent with studies comparing healthy and disease auditory and behavioral modifications within bee hives, and the embedding model is capturing some of the differences in acoustic profiles between the disease severity classes (Kazlauskas et al. 2016 Robles-Guerrero et 2017).
the peak at 645 Hz broadening and increasing in
CONCLUSION
We have shown for the first time the potential using custom deep learning audio models for predicting strengths of beehives based on multi-modal data composed of audio spectrograms and environment sensing data, despite collecting data across 26 sensors 26 hives without cross-sensor calibration and ground truth uncertainties for downstream prediction tasks. We believe that properly calibrated microphones with improved signal, better sensor placement, and better managed inspec- tion labels would further improve the predictions, reducing the need for frequent human inspections, a major limitation of current beekeeping practices. We have also shown the usefulness of semi- framework for modeling beehive audio, of which is easy to collect large
supervised learning amounts of samples but difficult to assign inspections, making annotation collection only possible during data collection. The learned latent space allows us to characterize different sound profiles and how they evolve over time, as shown with the progression in disease severity of pollinators in this work. Future efforts may explore how similar frameworks can be used to study the dynam- ics of beehives across longer time horizons combined with geographic data, and explore its use in forecasting, which can be used to improve labor and time management commercial beekeeping.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
David Anderson and Pietro Perona. Toward a science of computational ethology. Neuron, 84(1):
18-31, 2014.
Ada D. Rothschld and Guy Bloch. Differences in the sleep architecture of forager and young honeybees (apis mellifera). Journal of Experimental Biology, 211 2008. ISSN 0022-0949. doi: 10.12422jeb.016915. URL [tl.:/tees//ee..b00llaassssso/.. 211/15/2408)
S. Ferrari, M. Silva, M. Guarino, and D. Berekmans. Monitoring of swarming sounds in bee hives for early detection of the swarming period. Computers and Electronics Agriculture, 64(1):72 77, 2008. ISSN 0168-1699. doi: http:///ci.rrg/0000060...ompppp
05.010. URL |RL.ttpp:/ http:// 699900000
www. sci enced rect. com wttp:///c////ccc/ecceccaeeaaaaaa ottt://///oc/o...ooo///////////.
Smart
Sensors in precision livestock farming.
D. Howard, Olga Duran, Gordon Hunter, and Krzysztof Stebel. Signal processing the acoustics of honeybees (apis mellifera) to identify the "queenless" state in hives. Proceedings of the Institute of Acoustics, 35:290-299 297000999999000 01 2013.
Aren Jansen, Daniel PW Ellis, Shawn Hershey, R Channing Moore, Manoj Plakal, Ashok C Popat, and Rif A Saurous. Coincidence, categorization, and consolidation: Learning to recognize sounds with minimal supervision. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 121-125. IEEE, 2020.
Nadia Kazlauskas, M. Klappenbach, A. Depino, and F. Locatelli. Sickness behavior in honey bees. Frontiers in Physiology, 7, 2016.
Alexandra Klein, Bernard Vaissière, Jim Cane, Ingolf
Saul Cunningham, Claire Kremen, and Teja Tscharntke. Importance of pollinators in changing landscapes for world crops. Proceedings. Biological sciences / The Royal Society, 274:303 13, 03 2007. doi: 10. 1098//spb. 2006.3721.
Grzegorz Krzywoszyja, Ryszard Rybski, and Grzegorz Andrzejewski. Bee swarm detection based on comparison of estimated distributions samples of sound. IEEE Transactions on Instrumenta- tion and Measurement, 68( 37(10)3377-3784, 2018.
Kangkang Lu, Chuan-Sheng Foo, Kah Kuan Teh, Huy Dat Tran, and Vijay Ramaseshan Chan- drasekhar. Semi-Supervised Audio Classification with Consistency-Base Regularization. In Proc. Interspeech 2019, pp. 3654-3658, 2019. doi: 00214437/mmerppechh.0199999 2019-1231
URL
http://
doi http:////xxdoo..r//.....oo http:///.....11.//mmeeeeeeeeeeee
Margarita López- Uribe and Michael Memone-Finstrone Special issue: Honey bee research in the us: Current state and solutions to beekeeping problems. Insects, 10:22, 01 2019. doi: 10.3390/ insects 10010002..
Amro Qandour, Iftekhar Ahmad, Daryoush Habibi, and Mark Leppard. Remote beehive monitoring using acoustic signals. 2014.
Michael Ramsey, M. Bencsik, and Michael Newton. Extensive vibrational characterisation and long-term monitoring of honeybee dorso-ventral abdominal vibration signals. Scientific Reports, 8, 12 2018. doi: 10.
Richael-Thomas Ramsey, Martin Benesik. Michael Newton, Maritza Reyes, Maryline Pioz. Didier Crauser, Noa Simon-Delso, and Yves Le Conte. The prediction of swarming in honeybee colonies using vibrational spectra. Scientific Reports, 10, 12 2020. doi: 1038//4/ 1598 020-06010--655556566
Antonio Robles-Guerreto, Tonatiuh Saucedo-Anaya, Efrén González-Ramééez and Carlos Eric Galván-Tejada. Frequency analysis of honey bee buzz for automatic recognition of health sta- tus: A preliminary study. Res. Comput. Sci., 142:89 98, 2017.
Stephen Russell, Andrew B. Barron, and David Harris. Dynamic modelling of honey bee (apis mellifera) colony growth and failure. Ecological Modelling, 265:158-169, 2013. doi: https: //doi.org/10.1011 wwt.::colomee...2......m

---

Under review as a conference paper at ICLR 2021
F. Ruttner. Biogeography and Taxonomy of Honeybees. Springer-Verlag, Verlag, 1988. 9783540177814. URL baatps::/books..ooomt www/booksk.eeeeooooooo Picksssiiie 9 AACAAAAAAAA AAAAAAMMAA
ISBN
Shöichi F Sakagami and Hiromi Fukuda. Life tables for worker honeybees. Population Ecology, 10
(2):127-139, 1968.
Edward Southwick. The honey bee cluster as homeothermic superorganism. Comparative Biochemistry and Physiology Part A. Physiology, 75(4):641 645, 1983. ISSN 0300-9629. doi:
0ttp://000001000000000000000 0000660000000099000999999343444 6. URL http: sciencedi
com/ /tt.://caaa/aarteee/////03333333
Anton Stabentheiner, Helmut Kovac, and Robert Brodschneider.
colony thermoregulation-regnnn regulatory mechanisms and contribution of individuals dependence on age, location and thermal stress. PLoS one, ((1))e8967, 2010.
Honeybee
van der Sluijs and N. Vaage. Pollinators and global food security: the need for holistic global stewardship. Food Ethics, 1:75-91, 2016.
Milagra Weiss, Patrick Maes, William Fitz, Lucy Snyder, Tim Sheehan, Brendon Mott, and Kirk Anderson. Internal hive temperature as means of monitoring honey bee colony health migratory beekeeping operation before and during winter. Apidologie, 48, 05 2017. doi: 10.1007/ s13592-017-0512-8.
Mark L. Winston.
The Biology of the Honey Bee. First Harvard University Press, 1987. ISBN URL https: /btkks/ooos///eoooooo .owgoookksoooooo..oooooooooooooo google cww www../ooosssiiaa1ioiioi.........
9780674074095.
printsec-fronte www.tcooeerssoorceeeeeooooo gbs Batra__e___________aa_ae_ of
false
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.I DATA VALIDATION
We examine the quality of collected sensor data in several ways. For environmental sensor data, which composes of temperature, humidity, and pressure, developed dashboard to examine all datapoints to ensure smoothness across time and check all values to ensure they are within plausible range. For audio data, looked for features that could characterize mean activity across day. Bees have rigid circadian rhythms and are active during the day. By looking at simple metric such as were able see that the peak amplitude occurred average around midnight, when bees are mostly in the hive. This is congruent with established literature, which also suggests that bees often vibrate their wings modulate the hive temperature, which have verified through the inside temperature sensed for hives with bees compared to without bees (Weiss et al., 2017; Southwick 1983 Stabentheiner et al., 2010). Boxes with bees have consistent inside temperatures due to thermoregulation by healthy colonies, whereas an empty box we tested captures the same internal and external temperatures.
audio amplitude,
We also computed Pearson correlation coefficients between pairwise features to verify the plausi- bility of the linear relationships between our data features (Fig. []. and trained linear models to predict each feature based other features. We additionally examined conditional probabilities in order to verify each relationships in greater detail. summary, we find that there is some sensitivity differences across microphone, other sensor modalities were most likely self and cross-consistent. We also inspect audio spectrograms and remove data samples with failed audio captures.
Figure 7: Sampled training spectrograms (top) and reconstructions (bottom).
A.2 DATA PREPROCESSING
Each 56-second audio sample is converted into wav file and preprocessed with the python package Librosa to obtain a full sized mel spectrogram which 128x 1680 with a maximum frequency set 8192 Hz, half of the sampling rate of the data 16,384. This spectrogram is then down- sampled through mean pooling to 61 by 56, with 61 representing the frequency dimension, and 56 representing second-long time bins. Given the spectrogram captures frequencies well beyond bee sounds established in prior literature (Ramsey et al.) 2020 Howard et al., 2013 Ferrari et al. 2008), crop this spectrogram 56 dimensions, representing 56 by 56 downsampled spectrogram, which is used to feed into the embedding module after normalizing to between 0 and 1. We did not use some common transformations such as Mel-frequency cepstrum (MFCC), as enforces speech- dominant priors that do not apply to our data and would likely result in bias or data loss. We also normalize each environmental sensor feature separately and as well as prediction ground truths to between 0 and for more stable training.
A.3 DATASET PREPARATION
It can be difficult to assign data to each inspection given often inspections tend to correspond gaps in data collection. We worked with our inspector to understand the nature each inspection entry for each hive. For each inspection entry, match that entry the closest date have a full 96 samples for. We realized after doing this that number of inspections could not be matched to any sensor data, and unfortunately had to discard those labels from the dataset. In order to be stringent

---

Under review as a conference paper at ICLR 2021
1.00
Temp (Ext)
1
-0.7
-0.1
0.3
-0.1
-0.1
0.1
-0.09
-0.05
-0.1
0.75
Humidity (Ext)
Pressure (Ext)
-0.
0.2
-0.3
0.3
0.7
0.7
0.04
0.2
0.1
0.500
Temp (In)
0.3
0.2
0.3
1
0.07
0.02
0.02
0.2
0.2
0.09
Humidity (In)
-01
0.3
-0.07
0.3
0.2
0.1
0.00
Pressure (In)
0.1
0.02
0.3
0.5
0.03
-0.25
Audio Amplitude 0.1
0.03
0.7
0.02
0.3
0.5
1
0.2
0.3
0.1
Frames (Bees)
0..09
0.1
0.04
0.2
0.2
0.03
0.2
1
0.6
0.2
-0.50
Frames (Brood)
0.75
Disease Severity
01
0.3
0.1
0.09
0.1
0.09
0,1
0.2
0.2
-1.00
Temp (Ext)
Humniiity (Ext)
Pressure (Ert)
Temp (tm)
Humidity (in)
Pressure (In)
Audio Amplltude
Frames (Bees)
Frames (Brood)
Disezse Severity
Figure 8: Feature's Pearson correlation coefficients. Ext: external sensor, In: internal sensor. Aside from several expected strong correlations, there does not appear be other bivariate linear con- founders.
about label assignment, we only pair labels data if the difference in time is under days. This led to a dataset of 40 days with labels.
A.4 HYPERPARAMETERS
Training Adam was used as the optimizer for all objectives. We found that training with objec- tives was relatively stable, given sufficient pretraining. We used learning rate of 3e-5 for all except for disease classification, which we found required a slightly larger learning rate to converge at le- 4. The multiple objectives seems to have regularized the model from overfitting as evident in the validation curves, with the exception diseases likely because there is significant class imbalance and insufficient number of examples for diseased, due to the nature of the dataset. The number of pretrain iterations was determined empirically based on the decrease in validation loss. This pre- training is useful in order to prevent the network from overfiting the prediction loss from the very beginning and not learn to model the audio spectrogram.
Reconstruction
Frames
Disease Type
Disease Severtty
Negabbe Log Licitood
2.5
0.8
0-
2.0
:
2.
4000
0.5
2000
4200
6000
2000
0.0
2000
4000
6000
8000
2003 6000
4303
8000
2030
4000
6000
8050
Training
Figure 9: Tracking validation metrics for GPN across training iterations, averaged across 10-folds. Generative model pretraining occupies the bulk of the training iterations, after which the validation loss quickly plateaus.
Architecture We found that the predictor could overfit to the training set very easily, thus de- cided on using a single layer which attaches to multiple heads to allow for parameter sharing. We swept over different numbers of latent variables for the generative module, and found two latent
12

---

Under review as a conference paper at ICLR 2021
variables worked best when considering prediction performance and also reconstruction quality and interpretability of the latent space.
A.5 AUDIO FEATURES CAPTURED BY THE EMBEDDING MODEL
Magnitude After projecting audio data from the dataset into the embedding space across each day, and computing PCAs of each full-day latent representation, we found the first two PCA components captured significant portion of the variance of the embedding space, with PC-1 and PC-2 represent- ing respective 74. 17% and 10.61% % of the variance. To understand what specific audio features were captured by the embedding model, we swept ascending order across PC-1 and created 24 hour average frequency spectra for each sample. We observed the broadening of peak around 675 Hz and increase magnitude across all spectra. We plotted integrated magnitudes for frequency spec- tra from 0 to 8196 Hz against order PC-1, and found high correlation of magnitude within the defined region to position on PC-1, 1, demonstrating that the embedding model captured variation in magnitude across this frequency range.
And Magntaak
Aucrdd PC1
Aocending PC-3 WE
Figure 10: The average 24 frequency spectra for each of the 38 samples within the embedding space organized by ascending value on PC-1. Integrating the magnitude from 8196 Hz for each point and plotting against the order PC-1
13


Reviews:
[{"review_id": "vg5PVIvNHWS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes audio-driven multi-sensory modeling for predicting beehive strengths. The presentation is very clear; it describes what the task is, the data sources are, its limitations etc. And the proposed model based on GPN makes sense. Although, it seems the GPN may not outperform a baseline MLP from Table 1. \n\nOverall, this paper is a demonstration of utilizing machine learning methods and applying them in a plugin-and-test fashion to problems in natural science. Keeping the nature of the venue in mind; my main concern is the relevance and appropriateness of the technical content of this paper to ICLR -- i.e., the authors chose a wrong venue. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "mc9t_77OIKM", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors have implemented semi-supervised model for inferring beehive characteristics (frames, disease severity, and disease type) based on audio sensor and environmental sensor data. \n\nContributions:\n1. Sensor and environmental data collection from beehives together with sparse inspection data. This may be important contribution to the domain, but not a direct contribution to ICLR.\n2. Creating and training semi-supervised model to infer beehive characteristics based on low quantity of data in general and sparse inspection data. In particular, authors propose to use embedding module that is trained for reconstruction. The approach may be applicable to other domains with low quantity of audio data.\n\nI am not an expert in beehive monitoring, so it is hard for me to assess the novelty of the work done as specific to this domain.\nLooking from general point of view, the approach seems reasonable while taking into account low amount of data available and even lower amount of inspection data. Although I understand why only low amounts of data are available, the lack of data still raises questions whether the model is accurate enough and useful without larger amounts of data. And if more data were available, should the model architecture change significantly to achieve better accuracy?\n\nSpecific questions and concerns:\n1. Is there a reason why hive placement is discussed in \"inspections\" section? I think that placement is independent of inspections.\n2. I found the figures to be illustrative, but not very informative. For example, Figure 3 does not tell much except that some predictions are close to actuals and some are quite off. However, this is not really informative. Since the paper is handling a vertical domain, in my opinion, there should be discussion of precision vs. recall and possibly the cost of having false positives vs. false negatives. If you have false positive for disease, the cost would be additional human inspection, while if you have false negative, the cost might be hive loss. I acknowledge that there might be difficulties in addressing this since authors have very few inspection observations.\n3. Related to point 2: in Table 1, performance for disease severity and frames are shown by providing Huber loss. It is not clear how that relates into real world metrics that someone monitoring hives would consider. How does Huber loss translate into accuracy or accounting for false positives and false negatives?\n4. Figure 4 is not referenced and not discussed in the text.\n5. Figure 5 is also illustrative for eyeballing the similarities/differences, but it would be nice to have a quantitative evaluation of similarity possibly across frequencies.\n6. Figure 6 right hand side: again it is nice for eyeballing. It seems that latent 2 output is the biggest differentiator between healthy and unhealthy hives. Perhaps this should be investigated and mentioned. Audio outputs seem to be less useful. I would be concerned a bit that eyeballing may mislead that problem can be solved by just looking at the picture and that picture contains enough info for classification.\n7. What happens to prediction if audio is excluded and only environmental sensors are used for prediction? Is the accuracy very bad? \n\nIn summary, this is a nice proof-of-concept work in a specific vertical domain. It may have some lessons for other domains, but I am not sure it is strong enough for accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "O4AB40ZN5d1", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a semi-supervised model to predict the vitality of beehives. The inputs of the model are data from sensors (audio on one hand and environmental on the other hand such as temperature, humidity ...).  The objective is to predict simultaneously 3 values of interest: the frames state of beehives, the potential diseases and their severity. The architecure is composed of two modules, the first one is an auto-encoder in charge of embedding the audio spectrogram in a low latent dimensional space and the second one a MLP to predict the outputs from the latent spectrogram and the environmental data. The paper presents results of the proposed architecture on a small dataset, an ablation study to show the benefits of the auto-encoder module and the role of the environmental data and a latent space analysis to understand the ability of the model to capture relevant audio information linked to the diseases. \n\nIn my opinion, the paper is off topic. The presented model is very simple with nothing innovative (except for the application to beehives), the ML aspect of the paper is quite small (just 1 reference is ML oriented, all other references are from biology field) and the choice of the model is not really explained (why not something sequential ?), the experimental data are very small and it is hard to judge the relevance of the approach (only ablation studies is presented, no other approaches/baselines are considerd), I do not see how the approach can be generalized to other similar applications (with multi-modal data, audio combined to environmental time series). I am sure that the subject is interesting and important but I don't think that ICLR is the right conference for this paper. \n\nIn addition, the experimental part is hard to follow : the outputs of the model and the human labeling information are not clearly stated (what are the different diseases type ?  it is the same with diseases status ? the diseases severity seems to be discrete - bottom of page 3 - but a continuous Huber loss is used ?). The results are also very quickly analysed : Fig. 3 gives the impression that disease severity is always under estimated when the severity is high, but no analysis is provided. In summary, I can not state if the problem that the authors addres is a hard one or with simpler models same results can be achieved. \n\nMinor remarks : add color to figures 1,4 (and not black vs gray) for more visibility. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All Reviewers and myself believe that ICLR may not be the right venue for this paper. Hence, my recommendation is to REJECT it. As a brief summary, I highlight below some pros and cons that arose during the review and meta-review processes.

Pros:
- Important domain, but out of scope of ICLR.
- Collection of sensor and environmental data, which may be potentially hard to collect.

Cons:
- Not the appropriate venue.
- Lack of machine learning novelty.
- Potential lack of generalization of the proposed approach.
- Experimental part is hard to follow.
- Not very informative figures.
Decision: Reject