Introduction
Autonomous robots, such as service robots, operating in the human living environment with humans have to be able to perform various tasks and language communication. To this end, robots are required to acquire novel concepts and vocabulary on the basis of the information obtained from their sensors, e.g., laser sensors, microphones, and cameras, and recognize a variety of objects, places, and situations in an ambient environment. Above all, we consider it important for the robot to learn the names that humans associate with places in the environment and the spatial areas corresponding to these names; i.e., the robot has to be able to understand words related to places. Therefore, it is important to deal with considerable uncertainty, such as the robot's movement errors, sensor noise, and speech recognition errors.
Several studies on language acquisition by robots have assumed that robots have no prior lexical knowledge. These studies differ from speech recognition studies based on a large vocabulary and natural language processing studies based on lexical, syntactic, and semantic knowledge BIBREF0 , BIBREF1 . Studies on language acquisition by robots also constitute a constructive approach to the human developmental process and the emergence of symbols.
The objectives of this study were to build a robot that learns words related to places and efficiently utilizes this learned vocabulary in self-localization. Lexical acquisition related to places is expected to enable a robot to improve its spatial cognition. A schematic representation depicting the target task of this study is shown in Fig. FIGREF3 . This study assumes that a robot does not have any vocabularies in advance but can recognize syllables or phonemes. The robot then performs self-localization while moving around in the environment, as shown in Fig. FIGREF3 (a). An utterer speaks a sentence including the name of the place to the robot, as shown in Fig. FIGREF3 (b). For the purposes of this study, we need to consider the problems of self-localization and lexical acquisition simultaneously.
When a robot learns novel words from utterances, it is difficult to determine segmentation boundaries and the identity of different phoneme sequences from the speech recognition results, which can lead to errors. First, let us consider the case of the lexical acquisition of an isolated word. For example, if a robot obtains the speech recognition results “aporu”, “epou”, and “aqpuru” (incorrect phoneme recognition of apple), it is difficult for the robot to determine whether they denote the same referent without prior knowledge. Second, let us consider a case of the lexical acquisition of the utterance of a sentence. For example, a robot obtains a speech recognition result, such as “thisizanaporu.” The robot has to necessarily segment a sentence into individual words, e.g., “this”, “iz”, “an”, and “aporu”. In addition, it is necessary for the robot to recognize words referring to the same referent, e.g., the fruit apple, from among the many segmented results that contain errors. In case of Fig. FIGREF3 (c), there is some possibility of learning names including phoneme errors, e.g., “afroqtabutibe,” because the robot does not have any lexical knowledge. On the other hand, when a robot performs online probabilistic self-localization, we assume that the robot uses sensor data and control data, e.g., values obtained using a range sensor and odometry. If the position of the robot on the global map is unclear, the difficulties associated with the identification of the self-position by only using local sensor information become problematic. In the case of global localization using local information, e.g., a range sensor, the problem that the hypothesis of self-position is present in multiple remote locations, frequently occurs, as shown in Fig. FIGREF3 (d).
In order to solve the abovementioned problems, in this study, we adopted the following approach. An utterance is recognized as not a single phoneme sequence but a set of candidates of multiple phonemes. We attempt to suppress the variability in the speech recognition results by performing word discovery taking into account the multiple candidates of speech recognition. In addition, the names of places are learned by associating with words and positions. The lexical acquisition is complemented by using certain particular spatial information; i.e., this information is obtained by hearing utterances including the same word in the same place many times. Furthermore, in this study, we attempt to address the problem of the uncertainty of self-localization by improving the self-position errors by using a recognized utterance including the name of the current place and the acquired spatial concepts, as shown in Fig. FIGREF3 (e).
In this paper, we propose nonparametric Bayesian spatial concept acquisition method (SpCoA) on basis of unsupervised word segmentation and a nonparametric Bayesian generative model that integrates self-localization and a clustering in both words and places. The main contributions of this paper are as follows:
The remainder of this paper is organized as follows: In Section SECREF2 , previous studies on language acquisition and lexical acquisition relevant to our study are described. In Section SECREF3 , the proposed method SpCoA is presented. In Sections SECREF4 and SECREF5 , we discuss the effectiveness of SpCoA in the simulation and in the real environment. Section SECREF6 concludes this paper.
Lexical acquisition
Most studies on lexical acquisition typically focus on lexicons about objects BIBREF0 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 . Many of these studies have not be able to address the lexical acquisition of words other than those related to objects, e.g., words about places.
Roy et al. proposed a computational model that enables a robot to learn the names of objects from an object image and spontaneous infant-directed speech BIBREF0 . Their results showed that the model performed speech segmentation, word discovery, and visual categorization. Iwahashi et al. reported that a robot properly understands the situation and acquires the relationship of object behaviors and sentences BIBREF2 , BIBREF3 , BIBREF4 . Qu & Chai focused on the conjunction between speech and eye gaze and the use of domain knowledge in lexical acquisition BIBREF6 , BIBREF7 . They proposed an unsupervised learning method that automatically acquires novel words for an interactive system. Qu & Chai's method based on the IBM translation model BIBREF11 estimates the word-entity association probability.
Nakamura et al. proposed a method to learn object concepts and word meanings from multimodal information and verbal information BIBREF9 . The method proposed in BIBREF9 is a categorization method based on multimodal latent Dirichlet allocation (MLDA) that enables the acquisition of object concepts from multimodal information, such as visual, auditory, and haptic information BIBREF12 . Araki et al. addressed the development of a method combining unsupervised word segmentation from uttered sentences by a nested Pitman-Yor language model (NPYLM) BIBREF13 and the learning of object concepts by MLDA BIBREF10 . However, the disadvantage of using NPYLM was that phoneme sequences with errors did not result in appropriate word segmentation.
These studies did not address the lexical acquisition of the space and place that can also tolerate the uncertainty of phoneme recognition. However, for the introduction of robots into the human living environment, robots need to acquire a lexicon related to not only objects but also places. Our study focuses on the lexical acquisition related to places. Robots can adaptively learn the names of places in various human living environments by using SpCoA. We consider that the acquired names of places can be useful for various tasks, e.g., tasks with a movement of robots by the speech instruction.
Simultaneous learning of places and vocabulary
The following studies have addressed lexical acquisition related to places. However, these studies could not utilize the learned language knowledge in other estimations such as the self-localization of a robot.
Taguchi et al. proposed a method for the unsupervised learning of phoneme sequences and relationships between words and objects from various user utterances without any prior linguistic knowledge other than an acoustic model of phonemes BIBREF1 , BIBREF14 . Further, they proposed a method for the simultaneous categorization of self-position coordinates and lexical learning BIBREF15 . These experimental results showed that it was possible to learn the name of a place from utterances in some cases and to output words corresponding to places in a location that was not used for learning.
Milford et al. proposed RatSLAM inspired by the biological knowledge of a pose cell of the hippocampus of rodents BIBREF16 . Milford et al. proposed a method that enables a robot to acquire spatial concepts by using RatSLAM BIBREF17 . Further, Lingodroids, mobile robots that learn a language through robot-to-robot communication, have been studied BIBREF18 , BIBREF19 , BIBREF20 . Here, a robot communicated the name of a place to other robots at various locations. Experimental results showed that two robots acquired the lexicon of places that they had in common. In BIBREF20 , the researchers showed that it was possible to learn temporal concepts in a manner analogous to the acquisition of spatial concepts. These studies reported that the robots created their own vocabulary. However, these studies did not consider the acquisition of a lexicon by human-to-robot speech interactions.
Welke et al. proposed a method that acquires spatial representation by the integration of the representation of the continuous state space on the sensorimotor level and the discrete symbolic entities used in high-level reasoning BIBREF21 . This method estimates the probable spatial domain and word from the given objects by using the spatial lexical knowledge extracted from Google Corpus and the position information of the object. Their study is different from ours because their study did not consider lexicon learning from human speech.
In the case of global localization, the hypothesis of self-position often remains in multiple remote places. In this case, there is some possibility of performing an incorrect estimation and increasing the estimation error. This problem exists during teaching tasks and self-localization after the lexical acquisition. The abovementioned studies could not deal with this problem. In this paper, we have proposed a method that enables a robot to perform more accurate self-localization by reducing the estimation error of the teaching time by using a smoothing method in the teaching task and by utilizing words acquired through the lexical acquisition. The strengths of this study are that learning of spatial concept and self-localization represented as one generative model and robots are able to utilize acquired lexicon to self-localization autonomously.
Spatial Concept Acquisition
We propose nonparametric Bayesian spatial concept acquisition method (SpCoA) that integrates a nonparametric morphological analyzer for the lattice BIBREF22 , i.e., latticelm, a spatial clustering method, and Monte Carlo localization (MCL) BIBREF23 .
Generative model
In our study, we define a position as a specific coordinate or a local point in the environment, and the position distribution as the spatial area of the environment. Further, we define a spatial concept as the names of places and the position distributions corresponding to these names.
The model that was developed for spatial concept acquisition is a probabilistic generative model that integrates a self-localization with the simultaneous clustering of places and words. Fig. FIGREF13 shows the graphical model for spatial concept acquisition. Table TABREF14 shows each variable of the graphical model. The number of words in a sentence at time INLINEFORM0 is denoted as INLINEFORM1 . The generative model of the proposed method is defined as equation ( EQREF11 -). DISPLAYFORM0
Then, the probability distribution for equation () can be defined as follows: DISPLAYFORM0
The prior distribution configured by using the stick breaking process (SBP) BIBREF24 is denoted as INLINEFORM0 , the multinomial distribution as INLINEFORM1 , the Dirichlet distribution as INLINEFORM2 , the inverse–Wishart distribution as INLINEFORM3 , and the multivariate Gaussian (normal) distribution as INLINEFORM4 . The motion model and the sensor model of self-localization are denoted as INLINEFORM5 and INLINEFORM6 in equations () and (), respectively.
This model can learn an appropriate number of spatial concepts, depending on the data, by using a nonparametric Bayesian approach. We use the SBP, which is one of the methods based on the Dirichlet process. In particular, this model can consider a theoretically infinite number of spatial concepts INLINEFORM0 and position distributions INLINEFORM1 . SBP computations are difficult because they generate an infinite number of parameters. In this study, we approximate a number of parameters by setting sufficiently large values, i.e., a weak-limit approximation BIBREF25 .
It is possible to correlate a name with multiple places, e.g., “staircase” is in two different places, and a place with multiple names, e.g., “toilet” and “restroom” refer to the same place. Spatial concepts are represented by a word distribution of the names of the place INLINEFORM0 and several position distributions ( INLINEFORM1 , INLINEFORM2 ) indicated by a multinomial distribution INLINEFORM3 . In other words, this model is capable of relating the mixture of Gaussian distributions to a multinomial distribution of the names of places. It should be noted that the arrows connecting INLINEFORM4 to the surrounding nodes of the proposed graphical model differ from those of ordinal Gaussian mixture model (GMM). We assume that words obtained by the robot do not change its position, but that the position of the robot affects the distribution of words. Therefore, the proposed generative process assumes that the index of position distribution INLINEFORM5 , i.e., the category of the place, is generated from the position of the robot INLINEFORM6 . This change can be naturally introduced without any troubles by introducing equation ( EQREF12 ).
Overview of the proposed method SpCoA
We assume that a robot performs self-localization by using control data and sensor data at all times. The procedure for the learning of spatial concepts is as follows:
An utterer teaches a robot the names of places, as shown in Fig. FIGREF3 (b). Every time the robot arrives at a place that was a designated learning target, the utterer says a sentence, including the name of the current place.
The robot performs speech recognition from the uttered speech signal data. Thus, the speech recognition system includes a word dictionary of only Japanese syllables. The speech recognition results are obtained in a lattice format.
Word segmentation is performed by using the lattices of the speech recognition results.
The robot learns spatial concepts from words obtained by word segmentation and robot positions obtained by self-localization for all teaching times. The details of the learning are given in SECREF23 .
The procedure for self-localization utilizing spatial concepts is as follows:
The words of the learned spatial concepts are registered to the word dictionary of the speech recognition system.
When a robot obtains a speech signal, speech recognition is performed. Then, a word sequence as the 1-best speech recognition result is obtained.
The robot modifies the self-localization from words obtained by speech recognition and the position likelihood obtained by spatial concepts. The details of self-localization are provided in SECREF35 .
The proposed method can learn words related to places from the utterances of sentences. We use an unsupervised word segmentation method latticelm that can directly segment words from the lattices of the speech recognition results of the uttered sentences BIBREF22 . The lattice can represent to a compact the set of more promising hypotheses of a speech recognition result, such as N-best, in a directed graph format. Unsupervised word segmentation using the lattices of syllable recognition is expected to be able to reduce the variability and errors in phonemes as compared to NPYLM BIBREF13 , i.e., word segmentation using the 1-best speech recognition results.
The self-localization method adopts MCL BIBREF23 , a method that is generally used as the localization of mobile robots for simultaneous localization and mapping (SLAM) BIBREF26 . We assume that a robot generates an environment map by using MCL-based SLAM such as FastSLAM BIBREF27 , BIBREF28 in advance, and then, performs localization by using the generated map. Then, the environment map of both an occupancy grid map and a landmark map is acceptable.
Learning of spatial concept
Spatial concepts are learned from multiple teaching data, control data, and sensor data. The teaching data are a set of uttered sentences for all teaching times. Segmented words of an uttered sentence are converted into a bag-of-words (BoW) representation as a vector of the occurrence counts of words INLINEFORM0 . The set of the teaching times is denoted as INLINEFORM1 , and the number of teaching data items is denoted as INLINEFORM2 . The model parameters are denoted as INLINEFORM3 . The initial values of the model parameters can be set arbitrarily in accordance with a condition. Further, the sampling values of the model parameters from the following joint posterior distribution are obtained by performing Gibbs sampling. DISPLAYFORM0
where the hyperparameters of the model are denoted as INLINEFORM0 . The algorithm of the learning of spatial concepts is shown in Algorithm SECREF23 .
The conditional posterior distribution of each element used for performing Gibbs sampling can be expressed as follows: An index INLINEFORM0 of the position distribution is sampled for each data INLINEFORM1 from a posterior distribution as follows: DISPLAYFORM0
An index INLINEFORM0 of the spatial concepts is sampled for each data item INLINEFORM1 from a posterior distribution as follows: DISPLAYFORM0
where INLINEFORM0 denotes a vector of the occurrence counts of words in the sentence at time INLINEFORM1 . A posterior distribution representing word probabilities of the name of place INLINEFORM2 is calculated as follows: DISPLAYFORM0
where variables with the subscript INLINEFORM0 denote the set of all teaching times. A word probability of the name of place INLINEFORM1 is sampled for each INLINEFORM2 as follows: DISPLAYFORM0
where INLINEFORM0 represents the posterior parameter and INLINEFORM1 denotes the BoW representation of all sentences of INLINEFORM2 in INLINEFORM3 . A posterior distribution representing the position distribution INLINEFORM4 is calculated as follows: DISPLAYFORM0
A position distribution INLINEFORM0 , INLINEFORM1 is sampled for each INLINEFORM2 as follows: DISPLAYFORM0
where INLINEFORM0 denotes the Gaussian–inverse–Wishart distribution; INLINEFORM1 , and INLINEFORM2 represent the posterior parameters; and INLINEFORM3 indicates the set of the teaching positions of INLINEFORM4 in INLINEFORM5 . A topic probability distribution INLINEFORM6 of spatial concepts is sampled as follows: DISPLAYFORM0
A posterior distribution representing the mixed weights INLINEFORM0 of the position distributions is calculated as follows: DISPLAYFORM0
A mixed weight INLINEFORM0 of the position distributions is sampled for each INLINEFORM1 as follows: DISPLAYFORM0
where INLINEFORM0 denotes a vector counting all the indices of the Gaussian distribution of INLINEFORM1 in INLINEFORM2 .
Self-positions INLINEFORM0 are sampled by using a Monte Carlo fixed-lag smoother BIBREF29 in the learning phase. The smoother can estimate self-position INLINEFORM1 and not INLINEFORM2 , i.e., a sequential estimation from the given data INLINEFORM3 until time INLINEFORM4 , but it can estimate INLINEFORM5 , i.e., an estimation from the given data INLINEFORM6 until time INLINEFORM7 later than INLINEFORM8 INLINEFORM9 . In general, the smoothing method can provide a more accurate estimation than the MCL of online estimation. In contrast, if the self-position of a robot INLINEFORM10 is sampled like direct assignment sampling for each time INLINEFORM11 , the sampling of INLINEFORM12 is divided in the case with the teaching time INLINEFORM13 and another time INLINEFORM14 as follows: DISPLAYFORM0
[tb] Learning of spatial concepts [1] INLINEFORM0 , INLINEFORM1 Localization and speech recognition INLINEFORM2 to INLINEFORM3 INLINEFORM4 BIBREF29 the speech signal is observed INLINEFORM5 add INLINEFORM6 to INLINEFORM7 Registering the lattice add INLINEFORM8 to INLINEFORM9 Registering the teaching time Word segmentation using lattices INLINEFORM10 BIBREF22 Gibbs sampling Initialize parameters INLINEFORM11 , INLINEFORM12 , INLINEFORM13 INLINEFORM14 to INLINEFORM15 INLINEFORM16 ( EQREF25 ) INLINEFORM17 ( EQREF26 ) INLINEFORM18 ( EQREF28 ) INLINEFORM19 ( EQREF30 ) INLINEFORM20 ( EQREF31 ) INLINEFORM21 ( EQREF33 ) INLINEFORM22 to INLINEFORM23 INLINEFORM24 ( EQREF34 ) INLINEFORM25
Self-localization of after learning spatial concepts
A robot that acquires spatial concepts can leverage spatial concepts to self-localization. The estimated model parameters INLINEFORM0 and a speech recognition sentence INLINEFORM1 at time INLINEFORM2 are given to the condition part of the probability formula of MCL as follows: DISPLAYFORM0
When the robot hears the name of a place spoken by the utterer, in addition to the likelihood of the sensor model of MCL, the likelihood of INLINEFORM0 with respect to a speech recognition sentence is calculated as follows: DISPLAYFORM0
The algorithm of self-localization utilizing spatial concepts is shown in Algorithm SECREF35 . The set of particles is denoted as INLINEFORM0 , the temporary set that stores the pairs of the particle INLINEFORM1 and the weight INLINEFORM2 , i.e., INLINEFORM3 , is denoted as INLINEFORM4 . The number of particles is INLINEFORM5 . The function INLINEFORM6 is a function that moves each particle from its previous state INLINEFORM7 to its current state INLINEFORM8 by using control data. The function INLINEFORM9 calculates the likelihood of each particle INLINEFORM10 using sensor data INLINEFORM11 . These functions are normally used in MCL. For further details, please refer to BIBREF26 . In this case, a speech recognition sentence INLINEFORM12 is obtained by the speech recognition system using a word dictionary containing all the learned words. [tb] Self-localization utilizing spatial concepts [1] INLINEFORM13 INLINEFORM14 INLINEFORM15 INLINEFORM16 to INLINEFORM17 INLINEFORM18 () INLINEFORM19 () the speech signal is observed INLINEFORM20 add INLINEFORM21 to INLINEFORM22 INLINEFORM23 to INLINEFORM24 draw INLINEFORM25 with probability INLINEFORM26 add INLINEFORM27 to INLINEFORM28 INLINEFORM29
Experiment I
In this experiment, we validate the evidence of the proposed method (SpCoA) in an environment simulated on the simulator platform SIGVerse BIBREF30 , which enables the simulation of social interactions. The speech recognition is performed using the Japanese continuous speech recognition system Julius BIBREF31 , BIBREF32 . The set of 43 Japanese phonemes defined by Acoustical Society of Japan (ASJ)'s speech database committee is adopted by Julius BIBREF31 . The representation of these phonemes is also adopted in this study. The Julius system uses a word dictionary containing 115 Japanese syllables. The microphone attached on the robot is SHURE's PG27-USB. Further, an unsupervised morphological analyzer, a latticelm 0.4, is implemented BIBREF22 .
In the experiment, we compare the following three types of word segmentation methods. A set of syllable sequences is given to the graphical model of SpCoA by each method. This set is used for the learning of spatial concepts as recognized uttered sentences INLINEFORM0 .
The remainder of this section is organized as follows: In Section SECREF43 , the conditions and results of learning spatial concepts are described. The experiments performed using the learned spatial concepts are described in Section SECREF49 to SECREF64 . In Section SECREF49 , we evaluate the accuracy of the phoneme recognition and word segmentation for uttered sentences. In Section SECREF56 , we evaluate the clustering accuracy of the estimation results of index INLINEFORM0 of spatial concepts for each teaching utterance. In Section SECREF60 , we evaluate the accuracy of the acquisition of names of places. In Section SECREF64 , we show that spatial concepts can be utilized for effective self-localization.
Learning of spatial concepts
We conduct this experiment of spatial concept acquisition in the environment prepared on SIGVerse. The experimental environment is shown in Fig. FIGREF45 . A mobile robot can move by performing forward, backward, right rotation, or left rotation movements on a two-dimensional plane. In this experiment, the robot can use an approximately correct map of the considered environment. The robot has a range sensor in front and performs self-localization on the basis of an occupancy grid map. The initial particles are defined by the true initial position of the robot. The number of particles is INLINEFORM0 .
The lag value of the Monte Carlo fixed-lag smoothing is fixed at 100. The other parameters of this experiment are as follows: INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , INLINEFORM3 , INLINEFORM4 , INLINEFORM5 , INLINEFORM6 , INLINEFORM7 , and INLINEFORM8 . The number of iterations used for Gibbs sampling is 100. This experiment does not include the direct assignment sampling of INLINEFORM9 in equation ( EQREF34 ), i.e., lines 22–24 of Algorithm SECREF23 are omitted, because we consider that the self-position can be obtained with sufficiently good accuracy by using the Monte Carlo smoothing. Eight places are selected as the learning targets, and eight types of place names are considered. Each uttered place name is shown in Fig. FIGREF45 . These utterances include the same name in different places, i.e., “teeburunoatari” (which means near the table in English), and different names in the same place, i.e., “kiqchiN” and “daidokoro” (which mean a kitchen in English). The other teaching names are “geNkaN” (which means an entrance or a doorway in English); “terebimae” (which means the front of the TV in English); “gomibako” (which means a trash box in English); “hoNdana” (which means a bookshelf in English); and “sofaamae” (which means the front of the sofa in English). The teaching utterances, including the 10 types of phrases, are spoken for a total of 90 times. The phrases in each uttered sentence are listed in Table TABREF46 .
The learning results of spatial concepts obtained by using the proposed method are presented here. Fig. FIGREF47 shows the position distributions learned in the experimental environment. Fig. FIGREF47 (top) shows the word distributions of the names of places for each spatial concept, and Fig. FIGREF47 (bottom) shows the multinomial distributions of the indices of the position distributions. Consequently, the proposed method can learn the names of places corresponding to each place of the learning target. In the spatial concept of index INLINEFORM0 , the highest probability of words was “sofamae”, and the highest probability of the indices of the position distribution was INLINEFORM1 ; therefore, the name of a place “sofamae” was learned to correspond to the position distribution of INLINEFORM2 . In the spatial concept of index INLINEFORM3 , “kiqchi” and “daidokoro” were learned to correspond to the position distribution of INLINEFORM4 . Therefore, this result shows that multiple names can be learned for the same place. In the spatial concept of index INLINEFORM5 , “te” and “durunoatari” (one word in a normal situation) were learned to correspond to the position distributions of INLINEFORM6 and INLINEFORM7 . Therefore, this result shows that the same name can be learned for multiple places.
Phoneme recognition accuracy of uttered sentences
We compared the performance of three types of word segmentation methods for all the considered uttered sentences. It was difficult to weigh the ambiguous syllable recognition and the unsupervised word segmentation separately. Therefore, this experiment considered the positions of a delimiter as a single letter. We calculated the matching rate of a phoneme string of a recognition result of each uttered sentence and the correct phoneme string of the teaching data that was suitably segmented into Japanese morphemes using MeCab, which is an off-the-shelf Japanese morphological analyzer that is widely used for natural language processing. The matching rate of the phoneme string was calculated by using the phoneme accuracy rate (PAR) as follows: DISPLAYFORM0
The numerator of equation ( EQREF52 ) is calculated by using the Levenshtein distance between the correct phoneme string and the recognition phoneme string. INLINEFORM0 denotes the number of substitutions; INLINEFORM1 , the number of deletions; and INLINEFORM2 , the number of insertions. INLINEFORM3 represents the number of phonemes of the correct phoneme string.
Table TABREF54 shows the results of PAR. Table TABREF55 presents examples of the word segmentation results of the three considered methods. We found that the unsupervised morphological analyzer capable of using lattices improved the accuracy of phoneme recognition and word segmentation. Consequently, this result suggests that this word segmentation method considers the multiple hypothesis of speech recognition as a whole and reduces uncertainty such as variability in recognition by using the syllable recognition results in the lattice format.
Estimation accuracy of spatial concepts
We compared the matching rate with the estimation results of index INLINEFORM0 of the spatial concepts of each teaching utterance and the classification results of the correct answer given by humans. The evaluation of this experiment used the adjusted Rand index (ARI) BIBREF33 . ARI is a measure of the degree of similarity between two clustering results.
Further, we compared the proposed method with a method of word clustering without location information for the investigation of the effect of lexical acquisition using location information. In particular, a method of word clustering without location information used the Dirichlet process mixture (DPM) of the unigram model of an SBP representation. The parameters corresponding to those of the proposed method were the same as the parameters of the proposed method and were estimated using Gibbs sampling.
Fig. FIGREF59 shows the results of the average of the ARI values of 10 trials of learning by Gibbs sampling. Here, we found that the proposed method showed the best score. These results and the results reported in Section SECREF49 suggest that learning by uttered sentences obtained by better phoneme recognition and better word segmentation produces a good result for the acquisition of spatial concepts. Furthermore, in a comparison of two clustering methods, we found that SpCoA was considerably better than DPM, a word clustering method without location information, irrespective of the word segmentation method used. The experimental results showed that it is possible to improve the estimation accuracy of spatial concepts and vocabulary by performing word clustering that considered location information.
Accuracy of acquired phoneme sequences representing the names of places
We evaluated whether the names of places were properly learned for the considered teaching places. This experiment assumes a request for the best phoneme sequence INLINEFORM0 representing the self-position INLINEFORM1 for a robot. The robot moves close to each teaching place. The probability of a word INLINEFORM2 when the self-position INLINEFORM3 of the robot is given, INLINEFORM4 , can be obtained by using equation ( EQREF37 ). The word having the best probability was selected. We compared the PAR with the correct phoneme sequence and a selected name of the place. Because “kiqchiN” and “daidokoro” were taught for the same place, the word whose PAR was the higher score was adopted.
Fig. FIGREF63 shows the results of PAR for the word considered the name of a place. SpCoA (latticelm), the proposed method using the results of unsupervised word segmentation on the basis of the speech recognition results in the lattice format, showed the best PAR score. In the 1-best and BoS methods, a part syllable sequence of the name of a place was more minutely segmented as shown in Table TABREF55 . Therefore, the robot could not learn the name of the teaching place as a coherent phoneme sequence. In contrast, the robot could learn the names of teaching places more accurately by using the proposed method.
Self-localization that utilizes acquired spatial concepts
In this experiment, we validate that the robot can make efficient use of the acquired spatial concepts. We compare the estimation accuracy of localization for the proposed method (SpCoA MCL) and the conventional MCL. When a robot comes to the learning target, the utterer speaks out the sentence containing the name of the place once again for the robot. The moving trajectory of the robot and the uttered positions are the same in all the trials. In particular, the uttered sentence is “kokowa ** dayo”. When learning a task, this phrase is not used. The number of particles is INLINEFORM0 , and the initial particles are uniformly distributed in the considered environment. The robot performs a control operation for each time step.
The estimation error in the localization is evaluated as follows: While running localization, we record the estimation error (equation ( EQREF66 )) on the INLINEFORM0 plane of the floor for each time step. DISPLAYFORM0
where INLINEFORM0 denote the true position coordinates of the robot as obtained from the simulator, and INLINEFORM1 , INLINEFORM2 represent the weighted mean values of localization coordinates. The normalized weight INLINEFORM3 is obtained from the sensor model in MCL as a likelihood. In the utterance time, this likelihood is multiplied by the value calculated using equation ( EQREF37 ). INLINEFORM4 , INLINEFORM5 denote the INLINEFORM6 -coordinate and the INLINEFORM7 -coordinate of index INLINEFORM8 of each particle at time INLINEFORM9 . After running the localization, we calculated the average of INLINEFORM10 .
Further, we compared the estimation accuracy rate (EAR) of the global localization. In each trial, we calculated the proportion of time step in which the estimation error was less than 50 cm.
Fig. FIGREF68 shows the results of the estimation error and the EAR for 10 trials of each method. All trials of SpCoA MCL (latticelm) and almost all trials of the method using 1-best NPYLM and BoS showed relatively small estimation errors. Results of the second trial of 1-best NPYLM and the fifth trial of BoS showed higher estimation errors. In these trials, many particles converged to other places instead of the place where the robot was, based on utterance information. Nevertheless, compared with those of the conventional MCL, the results obtained using spatial concepts showed an obvious improvement in the estimation accuracy. Consequently, spatial concepts acquired by using the proposed method proved to be very helpful in improving the localization accuracy.
Experiment II
In this experiment, the effectiveness of the proposed method was tested by using an autonomous mobile robot TurtleBot 2 in a real environment. Fig. FIGREF70 shows TurtleBot 2 used in the experiments.
Mapping and self-localization are performed by the robot operating system (ROS). The speech recognition system, the microphone, and the unsupervised morphological analyzer were the same as those described in Section SECREF4 .
Learning of spatial concepts in the real environment
We conducted an experiment of the spatial concept acquisition in a real environment of an entire floor of a building. In this experiment, self-localization was performed using a map generated by SLAM. The initial particles are defined by the true initial position of the robot. The generated map in the real environment and the names of teaching places are shown in Fig. FIGREF73 . The number of teaching places was 19, and the number of teaching names was 16. The teaching utterances were performed for a total of 100 times.
Fig. FIGREF75 shows the position distributions learned on the map. Table TABREF76 shows the five best elements of the multinomial distributions of the name of place INLINEFORM0 and the multinomial distributions of the indices of the position distribution INLINEFORM1 for each index of spatial concept INLINEFORM2 .
Thus, we found that the proposed method can learn the names of places corresponding to the considered teaching places in the real environment. For example, in the spatial concept of index INLINEFORM0 , “torire” was learned to correspond to a position distribution of INLINEFORM1 . Similarly, “kidanokeN” corresponded to INLINEFORM2 in INLINEFORM3 , and “kaigihitsu” was corresponded to INLINEFORM4 in INLINEFORM5 . In the spatial concept of index INLINEFORM6 , a part of the syllable sequences was minutely segmented as “sohatsuke”, “N”, and “tani”, “guchi”. In this case, the robot was taught two types of names. These words were learned to correspond to the same position distribution of INLINEFORM7 . In INLINEFORM8 , “gomibako” showed a high probability, and it corresponded to three distributions of the position of INLINEFORM9 . The position distribution of INLINEFORM10 had the fourth highest probability in the spatial concept INLINEFORM11 . Therefore, “raqkukeN,” which had the fifth highest probability in the spatial concept INLINEFORM12 (and was expected to relate to the spatial concept INLINEFORM13 ), can be estimated as the word drawn from spatial concept INLINEFORM14 . However, in practice, this situation did not cause any severe problems because the spatial concept of the index INLINEFORM15 had the highest probabilities for the word “rapukeN” and the position distribution INLINEFORM16 than INLINEFORM17 . In the probabilistic model, the relative probability and the integrative information are important. When the robot listened to an utterance related to “raqkukeN,” it could make use of the spatial concept of index INLINEFORM18 for self-localization with a high probability, and appropriately updated its estimated self-location. We expected that the spatial concept of index INLINEFORM19 was learned as two separate spatial concepts. However, “watarirooka” and “kaidaNmae” were learned as the same spatial concept. Therefore, the multinomial distribution INLINEFORM20 showed a higher probability for the indices of the position distribution corresponding to the teaching places of both “watarirooka” and “kaidaNmae”.
The proposed method adopts a nonparametric Bayesian method in which it is possible to form spatial concepts that allow many-to-many correspondences between names and places. In contrast, this can create ambiguity that classifies originally different spatial concepts into one spatial concept as a side effect. There is a possibility that the ambiguity of concepts such as INLINEFORM0 will have a negative effect on self-localization, even though the self-localization performance was (overall) clearly increased by employing the proposed method. The solution of this problem will be considered in future work.
In terms of the PAR of uttered sentences, the evaluation value from the evaluation method used in Section SECREF49 is 0.83; this value is comparable to the result in Section SECREF49 . However, in terms of the PAR of the name of the place, the evaluation value from the evaluation method used in Section SECREF60 is 0.35, which is lower than that in Section SECREF60 . We consider that the increase in uncertainty in the real environment and the increase in the number of teaching words reduced the performance. We expect that this problem could be improved using further experience related to places, e.g., if the number of utterances per place is increased, and additional sensory information is provided.
Modification of localization by the acquired spatial concepts
In this experiment, we verified the modification results of self-localization by using spatial concepts in global self-localization. This experiment used the learning results of spatial concepts presented in Section SECREF71 . The experimental procedures are shown below. The initial particles were uniformly distributed on the entire floor. The robot begins to move from a little distance away to the target place. When the robot reached the target place, the utterer spoke the sentence containing the name of the place for the robot. Upon obtaining the speech information, the robot modifies the self-localization on the basis of the acquired spatial concepts. The number of particles was the same as that mentioned in Section SECREF71 .
Fig. FIGREF80 shows the results of the self-localization before (the top part of the figure) and after (the bottom part of the figure) the utterance for three places. The particle states are denoted by red arrows. The moving trajectory of the robot is indicated by a green dotted arrow. Figs. FIGREF80 (a), (b), and (c) show the results for the names of places “toire”, “souhatsukeN”, and “gomibako”. Further, three spatial concepts, i.e., those at INLINEFORM0 , were learned as “gomibako”. In this experiment, the utterer uttered to the robot when the robot came close to the place of INLINEFORM1 . In all the examples shown in the top part of the figure, the particles were dispersed in several places. In contrast, the number of particles near the true position of the robot showed an almost accurate increase in all the examples shown in the bottom part of the figure. Thus, we can conclude that the proposed method can modify self-localization by using spatial concepts.
Conclusion and Future Work
In this paper, we discussed the spatial concept acquisition, lexical acquisition related to places, and self-localization using acquired spatial concepts. We proposed nonparametric Bayesian spatial concept acquisition method SpCoA that integrates latticelm BIBREF22 , a spatial clustering method, and MCL. We conducted experiments for evaluating the performance of SpCoA in a simulation and a real environment. SpCoA showed good results in all the experiments. In experiments of the learning of spatial concepts, the robot could form spatial concepts for the places of the learning targets from human continuous speech signals in both the room of the simulation environment and the entire floor of the real environment. Further, the unsupervised word segmentation method latticelm could reduce the variability and errors in the recognition of phonemes in all the utterances. SpCoA achieved more accurate lexical acquisition by performing word segmentation using the lattices of the speech recognition results. In the self-localization experiments, the robot could effectively utilize the acquired spatial concepts for recognizing self-position and reducing the estimation errors in self-localization. As a method that further improves the performance of the lexical acquisition, a mutual learning method was proposed by Nakamura et al. on the basis of the integration of the learning of object concepts with a language model BIBREF34 , BIBREF35 . Following a similar approach, Heymann et al. proposed a method that alternately and repeatedly updates phoneme recognition results and the language model by using unsupervised word segmentation BIBREF36 . As a result, they achieved robust lexical acquisition. In our study, we can expect to improve the accuracy of lexical acquisition for spatial concepts by estimating both the spatial concepts and the language model.
Furthermore, as a future work, we consider it necessary for robots to learn spatial concepts online and to recognize whether the uttered word indicates the current place or destination. Furthermore, developing a method that simultaneously acquires spatial concepts and builds a map is one of our future objectives. We believe that the spatial concepts will have a positive effect on the mapping. We also intend to examine a method that associates the image and the landscape with spatial concepts and a method that estimates both spatial concepts and object concepts.
[] Akira Taniguchi received his BE degree from Ritsumeikan University in 2013 and his ME degree from the Graduate School of Information Science and Engineering, Ritsumeikan University, in 2015. He is currently working toward his PhD degree at the Emergent System Lab, Ritsumeikan University, Japan. His research interests include language acquisition, concept acquisition, and symbol emergence in robotics.
[] Tadahiro Taniguchi received the ME and PhD degrees from Kyoto University in 2003 and 2006, respectively. From April 2005 to March 2006, he was a Japan Society for the Promotion of Science (JSPS) research fellow (DC2) in the Department of Mechanical Engineering and Science, Graduate School of Engineering, Kyoto University. From April 2006 to March 2007, he was a JSPS research fellow (PD) in the same department. From April 2007 to March 2008, he was a JSPS research fellow in the Department of Systems Science, Graduate School of Informatics, Kyoto University. From April 2008 to March 2010, he was an assistant professor at the Department of Human and Computer Intelligence, Ritsumeikan University. Since April 2010, he has been an associate professor in the same department. He is currently engaged in research on machine learning, emergent systems, and semiotics.
[] Tetsunari Inamura received the BE, MS and PhD degrees from the University of Tokyo, in 1995, 1997 and 2000, respectively. He was a Researcher of the CREST program, Japanese Science and Technology Cooperation, from 2000 to 2003, and then joined the Department of Mechano-Informatics, School of Information Science and Technology, University of Tokyo as a Lecturer, from 2003 to 2006. He is now an Associate Professor in the Principles of Informatics Research Division, National Institute of Informatics, and an Associate Professor in the Department of Informatics, School of Multidisciplinary Sciences, Graduate University for Advanced Studies (SOKENDAI). His research interests include imitation learning and symbol emergence on humanoid robots, development of interactive robots through virtual reality and so on.

Question: How do they show that acquiring names of places helps self-localization?
Answer: unsupervised morphological analyzer capable of using lattices improved the accuracy of phoneme recognition and word segmentation, Consequently, this result suggests that this word segmentation method considers the multiple hypothesis of speech recognition as a whole and reduces uncertainty such as variability in recognition

Question: How do they evaluate how their model acquired words?
Answer: PAR score

Question: Which method do they use for word segmentation?
Answer: unsupervised word segmentation method latticelm

Question: Does their model start with any prior knowledge of words?
Answer: No