Keyword Spotter Model for Crop Pest and Disease Monitoring from Community Radio Data
Authors: Benjamin Akera
Joyce Nakatumba-Nabende
Ali Hussein
Daniel Ssendiwala
Jonathan Mukiibi
Venue:  (2020)
Keywords: keyword spotter
radio data
crop pest and disease
agriculture

Abstract: In societies with well developed internet infrastructure, social media is the leading medium of communication for various social issues especially for breaking news situations. In rural Uganda however, public community radio is still a dominant means for news dissemination. Community radio gives audience to the general public especially to individuals living in rural areas, and thus plays an important role in giving a voice to those living in the broadcast area. It is an avenue for participatory communication and a tool relevant in both economic and social development.This is supported by the rise to ubiquity of mobile phones providing access to phone-in or text-in talk shows. In this paper, we describe an approach to analysing the readily available community radio data with machine learning-based speech keyword spotting techniques. We identify the keywords of interest related to agriculture and build models to automatically identify these keywords from audio streams. Our contribution through these techniques is a cost-efficient and effective way to monitor food security concerns particularly in rural areas. Through keyword spotting and radio talk show analysis, issues such as crop diseases, pests, drought and famine can be captured and fed into an early warning system for stakeholders and policy makers.

Under review as a conference paper at ICLR 2020
KEYWORD SPOTTER MODEL FOR CROP PEST AND DIS- EASE MONITORING FROM COMMUNITY RADIO DATA
Anonymous authors Paper under double-blind review
ABSTRACT
In societies with well developed internet infrastructure, social media is the leading medium of communication for various social issues especially for breaking news situations. In rural Uganda however, public community radio is still dominant means for news dissemination. Community radio gives audience to the general public especially to individuals living in rural areas, and thus plays important role in giving a voice to those living in the broadcast area. participatory communication and tool relevant in both economic and social development. This is supported by the rise to ubiquity of mobile phones providing access phone-in or text-in talk shows. In this paper, describe approach analysing the readily available community radio data with machine learning-basd -based speech keyword spotting techniques. We identify the keywords of interest related agriculture and build models to automatically identify these keywords from audio streams. Our contribution through these techniques is cost-efficient and effective way to monitor food security concerns particularly in rural areas. Through keyword spotting and radio talk show analysis, issues such as crop diseases, pests, drought and famine can be captured and fed into early warning system for stakeholders and policy makers.
an avenue for
INTRODUCTION
Ensuring functional and near real-time system of surveillance for crop diseases and pests is of critical importance to sustaining the livelihoods of smallholder farmers in sub-Saharan Africa (Mutembesa al. 2018). Disease and pest surveillance systems have to be put in place to provide early warning to the farmers and the relevant agricultural research bodies. Usually, when crop disease or pest is reported in a given area, experts from the respective research institutes take time to reach the reported location to carry out investigations. This usually involves inspection of the crops at specific intervals (of about 10 km) along the more accessible main roads, covering only small proportions of the areas of interest in major districts (Mutembesa et al. 2018). Because the surveillance teams have to work within limited budgets, the surveys and the results from the surveys may be delayed or fewer regions may be sampled in particular year. As the health experts provide an annual snapshot of the health of crops across the country they are limited in their ability to provide real-time actionable surveillance data. In many cases, the farmers never get know the disease that has attacked their crops for weeks or even months.
In many areas in Uganda, the vast majority of the affected people will use social media communicate their concerns in their local communities. This social media is not Facebook or Twitter. its the local community radio stations existing in almost each village in Lab-Saharan Africa (Saeb et al. 2017). These rural radio stations give farmers an opportunity to interact with each other and also with the relevant agricultural authorities such as extension workers and agricultural experts. This can usually be through a number of formats like phone-in programs, live talk shows (Nakabugu) 2001). Specifically for some the radio stations, they have targeted agricultural talk shows that can host an expert from an agricultural research institute who can aim at providing specific information for example: about crop disease and pest management.
Keyword Spotting systems (KWS) is classification task that aims at detection and retrieving of series of words from a database audio streams. The advantage of using a KWS that unlike full automatic speech recognition systems, they can be developed without sufficient labelled data. This

---

Under review as a conference paper at ICLR 2020
is common especially in low resourced languages Menon et al. 2019). In this paper. discuss an implementation of a Keyword Spotting model that we use to mine local community radio content using specific keywords for low resourced language in Uganda. We evaluate our approach on the Luganda language which is a low-resource language that is currently spoken and used in many of the Agricultural communities in Uganda.
RELATED WORK
Previous work investigating crop disease surveillance utilizes different approaches. Some approaches have focused on setting up a crop disease surveillance network that relies on the use of mobile phones Mutembesa phones/Mutembesa disease detection aspect of the surveillance module uses computer vision and machine learning to detect plant diseases based leaf imaging Aduwo al. (2010); Mwebaze Owomugisha (2016).
et al.) (2018 2019) while others use satellite imagery [Zhang et al. (2044). The
Leaf-based approaches however rely on the use of imaging device for low-resource approaches Mutembbete
utilizing smartphones Quinn et al. (201 T); Quinn (2013); Mutembesa et al. (2018) 2019). This may
be limited in areas with no or low smartphone adoption.
Keyword spotting for low resource languages has been implemented before Menon et al. (001 ):
Saeb et al. (2017). The approaches include used include CNNs, siamese CNNs Bromley et al. (1994)
and autoencoders. Other models are designed for low computational resources such as Tang & Lin (2018) Coucke et al. (2019) due to the popularity of using keyword spotting to identify commands in smartphones, battery concerns from CPU requirements comes into play. Low resource languages pose a problem as models that consume a lot of data in training fail to converge due to low volume of text corpora and speech recording.
Luganda is almost zero-resource Bantu language, spoken in the central region of Uganda. Work on Luganda remains small compared to larger languages such as Kiwahili, Zulu and Hausa. Research on Luganda exists in machine translation Nandutu (2016), keyword spotting Menon et al. (2017) Saeb et al. 2017). Prior work on Luganda keyword spotting and radio monitoring of Luganda community radio has been initiated by Menon et al. (2017); Saeb et al. (201)) to generate insights concerning humanitarian aid and development. While radio content is publicly available and accessible with seemingly no data/privacy restrictions, there have been few interventions seeking to mine this data for surveillance purposes particularly for crop pests and diseases.
3 METHODOLOGY
3.1 BUILDING THE KEYWORD CORPUS
The primary source of keywords in this study are radio recordings captured from radio stations spread across Uganda. We selected 55 radio stations which commonly listened to in the central region. For each radio station, a google search was done to find out whether it had an online radio station and whether had its radio schedule available online. From the initial list of radio stations, 19 of them had online streams and of these at least 14 radios broadcast in Luganda. We identified radio schedules for 10 of these radio stations from the radio station websites and also by manually listening in to the stations. The purpose was to identify the time when there are talk shows particularly the agricultural ones. It was observed that for most talk shows topics of discussions are picked depending specific
on the audience demand, the trending topic in the society or country, sponsors/advertisers campaigns though still there are weekly talk shows which are focused on agriculture. The purpose was to identify the time when the talk shows were hired particularly the agricultural ones.
python script was written to stream the online stations at identified time. These were recorded as minute audio clips which were stored in shared Dropbox folder. The team also identified 2 radio stations which avail their radio content online for the past 7 days as 1-hour recordings. The websites were scrapped and the audio recordings were sorted depending on whether the hour was a talk show or not. Then the identified 1-hour audio clips that had talk shows were trimmed into 5 minute audio clips and these were added to the shared Dropbox folder.
2

---

Under review as a conference paper at ICLR 2020
Table 1: Aggregated Keywords from sample radio talkshow the fall army worm pest.
Sample Keywords
Keyword
Description
Frequency (per recording)
Akasanyi
Luganda term meaning worms
18 times
Obutunda
Luganda term meaning passion fruits
22 times
Kasooli
Luganda word meaning maize
35 times
The 5-minute audio clips were then played back and carefully listened to by a team of five volunteers with the purpose of identifying and extracting the commonly used agricultural terms that would be fed into the keyword spotting model.
To compliment the keywords captured from radio talkshows, we also scrapped an online local newspaper, For example, we obtained articles from a popular newspaper in Uganda commonly known as Bukedde One advantage of using online articles a source of keywords is that there are different ways in which the same crop disease or pest is mentioned and the spelling of such words can be to is captured specifically for Luganda. The keywords herbicides and general keywords. Translations were also added languages that is Luganda and English as well as an alternative keyword in form of a stem and this was in case the stem alone was unique keyword. (give example of keywords with the stemming).
then grouped into crops, diseases, fertilizers,
Both the keyword sets from the local radio and online sources were then aggregated into one keyword corpus of 193 keywords. An example of keywords extracted from radio talkshow on the Fall Army Worm pest affecting maize is shown in Table
3.2 SPEECH KEYWORD DATASET
Audio data collection was performed by crowdsourcing speech utterances of the different words from the keyword list. The use of studio captured samples seemed unrealistic, to mimic real-world settings the data was collected in a natural setting with noisy environments, poor quality recording equipment, and people talking in a natural, chatty way. Rather than using high quality microphones, and in formal setting. This was ensured through the audio data collection tool derived from Warden (2017) where the person speaking out the words can it using their phone or laptop wherever they are. In this study, we collected data from over 35 users who recorded the keywords in Luganda and English.
An important goal here was to record sufficient data to train the model but low enough allow for low-resource training. We ensured that we averaged 10 utterances per keyword. Keyword spotting models are much more useful if they are speaker independent, since the process of personalizing a model to individual requires intrusive user interface experience. With this in mind, the recording process had to be quick and easy to use, to reduce the number of people who would fail to complete it. The collected keyword audio data was encoded in ogg vorbis format.
3.3 DATA PREPROCESSING
3.3. ID CNN
In order to perform speech processing, our first step is to convert the recorded ogg keyword files to way files. As ogg lossy encoding format, used ffmpeg fim to decode the ogg vorbis files into wav audio files. The files were then transformed into 1d vectors using librosa librosa|McFec McFee et al. 2019).
Then for an audio signal w., with sampling rate and length t. We use the McFee et al. 2019 resampling feature below to normalize the sampling rate of all the samples to 8kHz as shown in Equation]
fre: : Wo + WgkH
(1)
https://ww..bkeee
3

---

Under review as a conference paper at ICLR 2020
3.3.2 SIAMESE CNN
For the Siamese CNN transform the ogg files into wav using ffmpeg fim Then we proceed generate spectograms and apply mel-compression.
MODEL ARCHITECTURE AND DESIGN
In this section, we briefly discuss the keyword spotting approaches that we use in this paper.
4.1 ID CONVOLUTION MODEL
In this study, We use 1-dimensional Convolutional Neural Network (CNN) which takes in as input the processed raw audio data. The input to the model is an array representing the audio waveform (X). The network designed to learn the set of parameters (0) to map the input to prediction (T") according to a hierarchical feature extraction given by equation 3
T= F(X = (1-1)))))00011110100000000001110
where L is the number of hidden layers in the network.
The final architecture created is 14-layer deep neural network with five 1D convolutional layers each with intermediate pooling layers. A dropout 0.5 was also applied after the two successive dense layers. The final layer softmax activation function to map to only 10 target keywords selected randomly from the Luganda/English corpus. The model was trained using batch gradient descent with the Adam Optimizer and a learning rate of 0.001.
4.2 SIAMESE CNN
The siamese CNN Bromley et al. (1994 takes as an input two mel-spectograms of size 32x100 We generate inputs the CNN by pairing mel-compressed spectograms of the same word and mel-compressed spectograms of different keywords. The inputs can be 'similar' being pronunciations of the same word or "different" being promunciations of different words.
4

---

Under review as a conference paper at ICLR 2020
Input1
Input2
Resnet50
Flatten
Flatten
Dense
Dense
L1 Distance
figureSiamese convolutional model
The siamese model performs binary classification at the output, classifying if the inputs similar or not based on an L1 distance metric. Binary cross-entropy was used as a loss function to achieve this task 3
- log()) (1 y) log(1 - -p)
RESULTS AND DISCUSSION
As a baseline experiment we trained baseline 5-layer densely connected network and our I convolutional model. We trained the model with early stopping at improvement in validation loss for 10 epochs, model stopped at 30 epochs. We trained our model on a total of 18426 samples and validated it on 4607 and tested on 5759 samples. This model was trained Colab. The evolution of loss and accuracy are shown Figure 1. Results are shown Table
K80 GPU Google
CONCLUSION AND FUTURE DIRECTION
Using ID convolutions for low-resource keyword spotting shows promising results. We aim to explore the usability of this work with other Bantu languages, specifically those geographically close to Luganda for example Runyankore, Kinyarwanda and Tooro as well as other linguistically further but geographically close languages such Luo and Karamojong. We also interested in the

---

Under review as a conference paper at ICLR 2020
Table 2: Node Classification Prediction Results
Model
Accuracy
Average Precision
Average Recall
F-1
Dense Baseline
0.62
0.62
0.60
0.60
Id-conv
0.93
0.93
0.93
0.93
Siamese CNN
0.85
0.92
0.75
0.83
model accuracy
model
0.9
Val
val
0.8
1.0
accurcy
0.8
0.6
0.4
0.5
10
spoch
epoch
Figure 1: Left: Model training and validation accuracy, Right: training and validation loss.
possibility of using transfer learning in keyword spotting in different dialects of the same language as in Acholi, Lango and Kumam for Luo and different but linguistically related languages as Luganda and Runyankore.
REFERENCES
Ffmpeg. http::/wwit ffwpeg org/
Jennifer R Aduwo, Ernest Mwebaze, and John A Quinn. Automated vision-based diagnosis of cassava mosaic disease. In Industrial Conference on Data Woriing-Worsspps,. pp. 114-122. New York, NY, 2010.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. In Advances in neural information processing systems, pp. 737-744, 1994.
Alice Coucke, Mohammed Chlich, Thibault Gisselbrecht, David Leroy, Mathieu Poumeyrol, and Thibaut Lavril. Efficient keyword spotting using dilated convolutions and gating. ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019. doi: 10.1109/icassp.2013 20198883474 URL http: /x www. aooi.org/100 1109/ /icasspppppppp
2019 9.
8483474
Brian McFee, Vincent Lostanlen, Matt MeCicar, Alexandros Metsai, Stefan Balke, Carl Thom, Colin Raffel, Dana Lee, Kyungyun Lee, Oriol Nieto, Jack Mason, Frank Zalkow, Dan Ellis, Eric Battenberg, Ryuichi Yamamoto, Rachel Bittner, Keunwoo Choi, Josh Moore, Ziyao Wei, Nullmightybofo, Pius Friesch, Fabian-Robert Stöter, Daro Here, Thassilo, Taewoon Kim, Matt Vollrath, Adam Weiss, CJ Carr, and Ajweiss-Dd. librosa/librosa: 0.7.0, 2019. URL https 7/zenodo wtw.:///.rggreeee/////eeee/
Raghav Menon, Armin Saeb, Hugh Cameron, William Kibira, John Quinn, and Thomas Niesler. Radio-browsing for developmental monitoring in uganda. In 20/7 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5795-5799, IEEE, 2017.
Raghav Menon, Herman Kamper, Ewald van der Westhuizen, John Quinn, and Thomas Niesler. Fea- ture exploration for almost zero-resouree asr-free keyword spotting using a multilingual bottleneck extractor and correspondence autoencoders. In INTERSPEECH, 2019.

---

Under review as a conference paper at ICLR 2020
Daniel Mutembesa, Christopher Omongo, and Ernest Mwebaze. Crowdsoureing real-time viral disease and pest information: case of nation-wide cassava disease surveillance in a developing country. In Sixth AAAI Conference on Human Computation and Crowdsourcing. 2018.
Daniel Mutembesa, Ernest Mwebaze, Solomon Nsumba, Christopher Omongo, and Humphrey Mutaasa. Mobile community sensing with smallholder farmers in a developing nation; scaled pilot for crop health monitoring. arXiv preprint arXiv: 1908. 07047, 2019.
Ernest Mwebaze and Godliver Owomugisha. Machine learning for plant disease incidence and severity measurements from leaf images. In 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 158-163. IEEE, 2016.
Sylvia Biraahwa Nakabugu. The role of rural radio in agricultural and rural development translating agricultural research information into messages for farm audiences. In First International Workshop on Farm Radio Broadcasting- 2001.
Irene Nandutu. Luganda text-to-speech machine. Master's thesis, School of Computing and Engi- neering, Uganda Technology and Management University, sep 2016.
John Quinn. Computational techniques for crop disease monitoring the developing world. In Ad- vances in Intelligent Data Analysis XII, pp. 13-18. Springer Berlin Heidelberg, 2013. doi: 10. 978-3-642-41398-- 2. URL http::ttp///tm doi /0070//0000000000000000000//---0
1007/99008178888 3 642 41398 2_
John Alexander Quinn, Kevin Leyton-Brown, and Ernest Mwebaze. Modeling and monitoring crop disease in developing countries. In Twenty-Fifth AAAI Conference Artificial Intelligence, 2011 1.
Armin Saeb, Raghav Menon, Hugh Cameron, William Kibira, John Quinn, and Thomas Niesler. Very low resource radio browsing for agile developmental and humanitarian monitoring. In INTERSPEECH, pp. 2118-2122, 2017.
Raphael Tang and Jimmy Lin. Deep residual learning for small-footprint keyword spotting. 2018
IEEE International Conference Acoustics, Speech and Signal Processsing (ICASSP), Apr 2018. doi: 10.1 10//iccssssss20111 8462688. URL htt.:////...............011.....
8462688
Pete Warden. Open speech recording.
https://gttnub..cm//eeewwwwr
open-speech- recordi-rcorrninggg 2017.
Jingcheng Zhang, Ruiliang Pu, Lin Yuan, Jihua Wang, Wenjiang Huang, and Guijun Yang. Monitoring powdery mildew of winter wheat by using moderate resolution multi-temporal satellite imagery.
PloS one, 94)7 2014.


Reviews:
[{"review_id": "SyeDlJt6FB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to design an effective and economical model which spots keywords about pests and disease from community radio data in Luganda and English. The author collected keywords from both radio and online newspapers, set up a Siamese convolutional neural network, which includes input 1 and input 2, Resnet 50, Flatten, five-layer dense and L1 distance, trained the model on a total of 18426 samples and validated on 4607 and tested on 5759 samples via a K80 GPU on Google Colab.  \n\nThe paper should be rejected because:\n(1) The generalizability to different use cases needs to be shown. The approach is too applied.\n(2) The fit to ICLR is not perfect. I would expect a stronger focus on representation learning.\n(3)\tModel\nThe paper lacks necessary explanation and clarification for the model, which is the core of the paper. For example, the part “Dense” in Siamese convolutional model is not well explained. The number of layers is up to 5, if there is no dropout, the model may have overfitting and gradient vanishing problems. \n(4)\tResult\nAs can be seen from both model accuracy and model loss curves in Figure 1, the part of 21st -30th epochs shows an increasing accuracy of training data while that of validation data does not increase anymore and a declining loss of training data while that of validation does not decrease. This is a tendency of overfitting, which means the model should adopt 20 epochs rather than 30 epochs. \n(5)\tIncompleteness \nCompared with the detailed depiction of the first three parts, part 4, part 5 and part 6 are insufficient. It is necessary to complement these three parts with interpretation, explanation and discussion. \n\nThings to improve the paper that did not impact the score:\n(1)\tLanguage problems such as in page 3 paragraph 2: “To compliment the keywords captured from radio talkshows, we also scrapped an online local newspaper.”  Here “compliment” should be “complement”, and “scrap” is semantically incorrect. \n(2)\tPage 4 part 4.1: “…according to a hierarchical feature extraction given by equation 3.” It should be equation 2. \n(3)\tFormat in references: fonts are not uniformed.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Skeu3xh0KH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes an approach to analyze radio data with ML-based speech keyword techniques. The authors identify keywords related to agriculture and build a model that can automatically detect these keywords of interest. Their contribution is the proposed model relies on relatively simple neural networks (15-layer 1-D CNNs) to achieve keyword detection of low resourced language (e.g., Luganda). Therefore, they are capable of monitoring food security concerns in rural areas.\n\nThis paper should be rejected because (1) no novel algorithm is proposed and only straightforward solution is used (2) the paper never clearly demonstrates the existence of problem they are trying to solve, (3) the experiments are difficult to understand and missing many details, and (4) the paper is incomplete and unpolished.\n\nMain argument\nThe paper does not do a great job of demonstrating that current model development and drawbacks of dealing with low resourced language (e.g., Luganda). They said they solve the problem by the Keywords Spotting system (KWS), but they did not give a comprehensive review of the current progress of KWS. For example, they should include and compare with the paper, called “Speech recognition and keyword spotting for low-resource languages: Babel Project Research at CUED” by Dr. Mark Gales.\n\nWhen it comes to methodology, the authors use unnecessary pages to describe how they collect Luganda audios and keyword dataset. It would be appropriate to illustrate their data flow or detection pipeline instead of irrelevant engineering details. In addition, SIAMESE CNN is used to compare performance, but the authors use a relatively large portion of the paper to explain its architecture. Did their method developed beyond SIAMESE CNN? Otherwise, it would be inappropriate to describe it in methodology and model design. For the 1D convolution model, the model is demonstrated in an extremely simple way. They did not even mention their target data (Y), please clarify it in detail.\nThe experiment did not provide convincing evidence of the utility of the proposed method (1-D convolution). They only demonstrated results in 1 dataset, which did not indicate the effectiveness of the model. There are so many missing details. How many times the experiment they did? How much size is an audio file? Why 1d-conv show the same values of accuracy, precision, and recall? What’s the model drawn in Figure 1? The paper lacks complete discussion on their empirical results. We did not know how to interpret their results.\n\nConsidering the above, there are many drawbacks to the paper. The paper seems incomplete. Therefore, we highly recommend rejecting the paper.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BklSdgXQtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview\n\nThis paper presents a very interesting application of speech keyword spotting techniques; the aim is to listen to continuous streams of community radio in Uganda in order to spot keywords of interest related to agriculture to monitor food security concerns in rural areas. The lack of internet infrastructure results in farmers in rural areas using community radio to share concerns related to agriculture. Therefore accurate keyword spotting techniques can potentially help researchers flag up areas of interest. The main engineering challenge that the study deals with is the fact that there isn’t a lot of training data available for languages like Lugandu.\n\nDetailed Comments\n\n Section 3\n\n1. The word “scrapped” should be replaced with “scraped”.\n2. Its not entirely clearly what is meant by “as well as an alternative keyword in form of a stem”. What is meant by “stem” in this context? Maybe a citation or an example would make it more clear. \n3. The corpus of keywords supposedly contains 193 distinct keywords however the models in Section 4 are only trained to discriminate between 10 randomly sampled keywords. I don’t understand why this is. Training on the full corpus would allow the network to see more training data and consequently might result in more accurate models. \n\nSection 4\n\n1. The authors refer to Equation 3 in Section 4.1, but I think the reference is to Equation 2.\n2. The 1-D CNN has 14 layers but it is not clear to me what the size of the final network is. It would be useful to provide more details of the architecture and a comment about the network size either in terms of number of parameters, number of computations or size of the network weights on disk. \n3. The figure for  the Siamese network should be numbered. \n4. The authors say that the inputs to the ResNets are of shape 32x100. What do these dimensions refer to? Are their 32 frequency bins? Are their 100 frames as inputs? What are the parameters of the FFT computation? Do 100 frames correspond to a second of audio? \n5. How many parameters are their in the Siamese network? \n6. Its not clear to me how the Siamese networks are used during inference. The authors say they use the L1 distance to determine if a given test example is similar to a given keyword. How are the examples for the keyword of interest selected? Are all training examples used and the scores average? Are the training examples averaged first to form a centroid vector for  the keyword? \n\nComments on the methodology\n\nOne of the main challenges in this work is the fact that there isn’t a large number of training examples available. However the authors still train relatively large acoustic models with 14 layers for 1-D CNN and a ResNet for the Siamese architecture. There are several studies for keyword spotting on mobile devices that aim to train tiny networks that consume very little power [1] [2]. I think it would be more appropriate to start with architectures similar to these due to the small size of the training dataset. Additionally, it would also be very useful to try and identify languages that are phonetically similar to Lugandu and that have training datasets available for speech recognition. Acoustic models trained on this data can then be adapted to the keyword spotting task using the training set collected for this task. Note that the languages need to have only a small amount of phonetic overlap in order for these acoustic models to be useful starting points for training keyword spotting models. \n\nComments on the evalation\n\nThe keyword spotting models presented here are trained with the intention of applying them to streaming radio data. However, the models are trained and tested on fixed chunks of audio. A big problem with this experimental design is that the models can overfit to confounding audio cues in the training data. For example, if all training data are in the form of 1 second audio chunks and all chunks have at least some silence at the start, then the models learn that a keyword is always preceded by a certain duration of silence. This is not the case when keywords occur in the middle of sentences in streaming audio data. The fact that the evaluation is also performed on individual chunks of audio fails to evaluate how the trained models would behave when presented streaming audio. I am certain that when applied to streaming audio these models would false trigger very often, however this fact isn’t reflected by measuring precision and recall on fixed chunks of test audio. \n\nA more principled evaluation strategy would be to present results in the form of detection-error tradeoff (DET) curves [1], [2]. Here the chunks with examples of the keyword in question can be used to measure the number of false rejects by the system. And long streams of audio data that do not contain the keyword should be used to measure the false alarms. Given that its relatively easy to collect streams of radio data, the false alarms can be measured in terms of the number of false alarms per unit time (minutes, hours or days). This evaluation strategy roughly simulates the streaming conditions in which this model is intended to be deployed. Furthermore, DET curves present the tradeoff between false alarms and false rejects as a function of the operating threshold, which provides much more insight into the performance/accuracy of the trained models. \n\nSummary\n\nI think the area of application of this work is extremely interesting, however the training and evaluation methodologies have to be updated in order to realistically measure the way this system might perform in real-world test conditions. \n\nReferences\n\n[1] Small-footprint Keyword Spotting Using Deep Neural Networks\n[2] Efficient Voice Trigger Detection for Low Resource Hardware\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main summary:  Design an effective and economical model which spots keywords about pests and disease from community radio data in Luganda and English.

Discussions:
all reviewers vote on rejecting the paper, due to lack of generalizability, training and evaluation discussion need work
Recommendation: Reject
Decision: Reject