Introduction
Ultrasound technology is a widespread technology in speech research for studying tongue movement and speech articulation BIBREF0 due to its attractive characteristics, such as imaging at a reasonably rapid frame rate, which empowers researchers to envision subtle and swift gestures of the tongue in real-time. Besides, ultrasound technology is portable, relatively affordable, and clinically safe and non-invasive BIBREF1. The mid-sagittal view is regularly adapted in ultrasound data as it displays relative backness, height, and the slope of various areas of the tongue. Quantitative analysis of tongue motion needs the tongue contour to be extracted, tracked, and visualized.
Manual frame-by-frame tongue contour extraction is a cumbersome, subjective, and error-prone task. Moreover, it is not a feasible solution for real-time applications. In conventional techniques, for extracting ultrasound tongue contours, a discrete set of vertices are first annotated near the lower part of the tongue dorsum defining initial deformable tongue contour BIBREF2. Then, through an iterative minimization process using features of the image, the annotated contour is regulated toward the tongue dorsum region. For instance, in active contour models technique (e.g., EdgeTrak software) BIBREF3, BIBREF4, two internal and external energy functions are minimized over the image gradient. The requirement of feature extraction for each image and accurate initialization are two main drawbacks for those classical techniques. Another alternative scenario is to use semi-supervised machine learning models for automatic segmentation of tongue contour regions. Then, tongue contours are extracted automatically using post-processing stages.
Semi-supervised machine learning-based methods BIBREF5 are first utilized for ultrasound tongue contour segmentation in an study by BIBREF6, BIBREF7 while deep learning models emerge in this field through studies by BIBREF0, BIBREF8. They fine-tuned one pre-trained decoder part of a Deep Belief Network (DBN) model to infer tongue contours from new instances. End-to-end fashion supervised deep learning techniques, outperformed previous techniques in recent years. For example, U-net BIBREF9 has been used for automatic ultrasound tongue extraction BIBREF10, BIBREF11. After successful results of deep learning methods, the focus of advanced techniques for tongue contour extraction is more on generalization and real-time performance BIBREF12, BIBREF13, BIBREF14.
Although deep learning methods have been utilized successfully in many studies, manual annotation of ultrasound tongue databases is still cumbersome, and the performance of supervised methods mostly depends on the accuracy of the annotated database as well as the number of available samples. Available databases in this field are annotated by linguistics experts for many years employing landmark points on the tongue contours. In this work, we proposed a new direction for the problem of ultrasound tongue contour extraction using a deep learning technique where instead of tracking the tongue surface, landmarks on the tongue are tracked. In this way, researchers can use previously available linguistics ultrasound tongue databases. Moreover, the whole process of tongue contour extraction is performed in one step, where it increases the performance speed without comprising accuracy or generalization ability of the previous techniques.
Methodology
Similar to facial landmark detection methods BIBREF15, we considered the problem of tongue contour extraction as a simple landmark detection and tracking. For this reason, we first developed a customized annotator software that can extract equally separated and randomized markers from segmented tongue contours in different databases. Meanwhile, the same software could fit B-spline curves on the extracted markers to revert the process for evaluation purposes.
To track landmarks on the tongue surface, we designed a light-version deep convolutional neural network named TongueNet. Figure FIGREF1 shows TongueNet architecture. In each layer, convolutional operations followed by ReLU activations as a non-linearity function as well as Batch normalization layers to improve the regularization, convergence, and accuracy. For the sake of better generalization ability of the model, in the last two layers, fully connected layers are equipped with Drop-out layers of $50\%$. To find the optimum number of required points in the output layer, we used the number of points ($\#$ in Figure FIGREF1) from 5 to 100 (see Figure FIGREF2 for samples of this experiment with 5, 10, 15, 20, 25, and 30 points as the output).
Experimental Results and Discussion
There is usually a trade-off between the number of training samples and the number of trainable parameters in a deep network model BIBREF16. In general, more data we have, better result are generated by supervised deep learning methods. Data augmentation helps to increase the number of training data, but a bigger dataset needs a better and most likely bigger network architecture in terms of generalization. Otherwise, the model might over-fitted or under-fitted on training data. Using our annotation software, we automatically extracted landmarks of 2000 images from the UOttawa database BIBREF14 had been annotated for image segmentation tasks. The database was randomly divided into three sets: 90$\%$ training, 5$\%$ validation, and 5$\%$ testing datasets. For testing, we also applied the TongueNet on the UBC database BIBREF14 without any training to see the generalization ability of the model. During the training process of TongueNet, we employed an online data augmentation, including rotation (-25 to 25 degrees), translation (-30 to 30 points in two directions), scaling (from 0.5 to 2 times), horizontal flipping, and combination of these transformations, and annotation point locations are also was transformed, correspondingly. From our extensive random search hyper-parameter tuning, learning rate, the number of iterations, mini-batch sizes, the number of epochs was determined as 0.0005, 1000, 30, 10, respectively. We deployed our experiments using Keras with TensorFlow as the backend on a Windows PC with Core i7, 4.2 GHz speed using one NVIDIA 1080 GPU unit, and 32 GB of RAM. Adam optimization with fixed momentum values of 0.9, was utilized for training.
We trained and tested TongueNet with a different number of points as the output size. We first evaluated the scenario of equally spaced landmarks on the tongue surface. In this method, we captured all the points in equal distances respect to their neighbors. Our results from selecting five points to the number of pixels in the horizontal axis of the image (image width) revealed that the results of equally spaced selected points are not significant. Figure FIGREF3 shows randomly selected frames from real-time tracking of TongueNet using ultrasound tongue landmarks. As can be seen from the figure, automatically selection of points as annotation, which is used in many previous studies (see BIBREF17 for an example), can not provide accurate results. In a separate study, we extract annotation points from the same database using a randomly spaced selection method on tongue contours. We add restrictions for points that they should be at a minimum distance from each other as well as omitting outliers from the database. We saw that the optimum number of points was ten points for our experiments. Figure FIGREF4 shows some randomly selected results from training TongueNet on a randomly selected point database. From the figure, better accuracy of the TongueNet can be seen qualitatively. Note that we didn't apply any image enhancement or cropping for databases.
To show the performance ability of TongueNet quantitatively, we first fitted B-spline curves using the OpenCV library on the instances of TongueNet. Then, we compared the value of the mean sum of distance (MSD) BIBREF8 for TongueNet, sUNET BIBREF11, UNET BIBREF18, BowNet BIBREF13, and IrisNet BIBREF14 deep learning models. From table TABREF6, it can be seen that TongueNet could reach to results similar to the state of the art deep learning models in this field. Note that there are some approximation errors for the curve-fitting procedure of the TongueNet and skeletonization process for extracting tongue contours from segmentation results of other deep learning models.
We even tested TongueNet on a new database from UBC (the same database used in BIBREF14) to evaluate the generalization ability of the landmark tracking technique. Although TongueNet has not trained on that database, it could predict favorable instances for video frames with different data distributions. This shows the capability of TongueNet for managing of the over-fitting. From Table. TABREF6, the difference of MSD values is not significant between models, while IrisNet could find better MSD value. However, in terms of speed, TongueNet outperforms other deep learning models while post-processing time is not considered for them.
We likewise tested TongueNet on a new database from the University of British Columbia (the same database used in BIBREF14) to evaluate the generalization ability of the landmark tracking technique. Although TongueNet has not trained on that database, it could predict favorable instances for that novel video frames with different data distributions. This shows the capability of TongueNet for managing of the over-fitting. From Table. TABREF6, although the difference of MSD values is not significant between models, IrisNet could find better MSD value. However, in terms of speed performance, TongueNet outperforms other deep learning models while post-processing time was not included for calculation of frame rates.
Conclusion and Discussion
In this paper, we presented TongueNet, a simple deep learning architecture, with a novel training approach for the problem of Tongue contour extraction and tracking. Unlike similar studies, we used several points to track on the tongue surface instead of the whole tongue contour region. In recent tongue contour tracking, for quantitative studies, a two-phase procedure is performed, including image segmentation and post-processing for extraction of tongue contour that this increase computational costs. Furthermore, available previously annotated tongue databases in articulation studies could not be utilized for deep learning techniques until now. Using TongueNet, we provided a new tool for this literature, while old databases now can be used for developing a better tongue contour tracking technique.
All landmark locations in this study are annotated automatically with two different approaches, and we experienced that data augmentation has a significant rule in the accuracy of TongueNet. From our experimental results, we can anticipate that if an extensive manually annotated database, might be a combination of several databases, is employed for training of a deep learning model such as TongueNet, the accuracy of the model would be boosted considerably. The materials of this study will help researchers in different fields such as linguistics to study tongue gestures in real-time easier, accessible, and with higher accuracy than previous methods. The current infant TongueNet technique needs to be developed, trained, and extended as a fast, accurate, real-time, automatic method applicable for available ultrasound tongue databases.

Question: How big are datasets used in experiments?
Answer: 2000 images

Question: What previously annotated databases are available?
Answer: the UBC database BIBREF14