{"id":61357,"date":"2016-07-19T15:16:00","date_gmt":"2016-07-19T19:16:00","guid":{"rendered":"https:\/\/www.nasa.gov\/?migration=converted-node-387591"},"modified":"2023-07-26T15:12:59","modified_gmt":"2023-07-26T19:12:59","slug":"collaborative-augmented-reality-with-hands-free-gesture-control-for-remote-astronaut-training-and-mentoring","status":"publish","type":"post","link":"https:\/\/www.nasa.gov\/directorates\/stmd\/space-tech-research-grants\/collaborative-augmented-reality-with-hands-free-gesture-control-for-remote-astronaut-training-and-mentoring\/","title":{"rendered":"Collaborative Augmented Reality with Hands-Free Gesture Control for Remote Astronaut Training and Mentoring"},"content":{"rendered":"\n<p><strong>Hiroshi Furuya<\/strong><br>\n<em>Columbia University<\/em><\/p>\n\n\n<div id=\"\" class=\"hds-media hds-module wp-block-image\"><div class=\"margin-left-auto margin-right-auto nasa-block-align-inline\"><div class=\"hds-media-wrapper margin-left-auto margin-right-auto\"><figure class=\"hds-media-inner hds-cover-wrapper hds-media-ratio-none \"><a href=\"https:\/\/www.nasa.gov\/wp-content\/uploads\/2016\/07\/furuya_hiroshi.jpg\"><img fetchpriority=\"high\" decoding=\"async\" width=\"500\" height=\"457\" src=\"https:\/\/www.nasa.gov\/wp-content\/uploads\/2016\/07\/furuya_hiroshi.jpg?w=500\" class=\"attachment-2048x2048 size-2048x2048\" alt=\"Hiroshi Furuya\" style=\"transform: scale(1); transform-origin: 50% 50%; object-position: 50% 50%; object-fit: cover;\" block_context=\"nasa-block\" srcset=\"https:\/\/www.nasa.gov\/wp-content\/uploads\/2016\/07\/furuya_hiroshi.jpg 500w, https:\/\/www.nasa.gov\/wp-content\/uploads\/2016\/07\/furuya_hiroshi.jpg?resize=300,274 300w, https:\/\/www.nasa.gov\/wp-content\/uploads\/2016\/07\/furuya_hiroshi.jpg?resize=400,366 400w\" sizes=\"(max-width: 500px) 100vw, 500px\" loading=\"eager\" \/><\/a><\/figure><figcaption class=\"hds-caption padding-y-2\"><div class=\"hds-caption-text p-sm margin-0\">Hiroshi Furuya<\/div><\/figcaption><\/div><\/div><\/div>\n\n\n<p>Augmented Reality (AR), where 3D and 2D images are overlaid on a user\u2019s natural field of view like a heads up display, is being developed for use in manned space applications. In the past 2 years, NASA, through AR-eProc, and the European Space Agency, through MARSOP, have demonstrated the potential of AR to improve hands-on training and just-in-time training (JITT).<\/p>\n\n\n\n<p>This research aims demonstrate the feasibility and test the performance of novel hands-free, remote or proximity, and instantaneous collaboration or mentoring between individuals in the AR environment to augment audio communication for training, task completion, and other space applications. Real-time AR collaboration can provide novel, flexible spontaneous communication capabilities not found in current systems. Collaboration would be facilitated by drawing \u009d on, or annotating, preexisting AR graphics using tablets or gesture detection which can also be viewed by other individuals in real time. Hands-free operation would be accomplished by utilizing radar-based gesture recognition transceivers which are environment-robust and embeddable in rigid materials. They precisely measure motion in 3D, allowing for gestures adapted to limited ranges of motion. Embedded gesture sensors and AR headsets enable potential expansion of use to EVA or other space suit applications. The research proposed aims to demonstrate the feasibility of the combined use of such systems and evaluate their performance.<\/p>\n\n\n\n<p>This would fulfill research in Technical Area 04: Robotics and Autonomous Systems Section 4 items 3 and 5 (4.4.3,5). The potential for AR control in EVA applications opens future development in AR communication with autonomous systems (4.4.3). This research by its remote collaborative nature can provide tools that facilitate resource and task allocation, trading and sharing of control, and dialogue<\/p>\n\n\n\n<p>management. \u009d Use Cases: Remote or proximity AR collaboration or mentoring on Earth, in deep sea, or in Low Earth Orbit. Proximity-only AR collaboration or mentoring in deep space or situations with low communication bandwidth to remote sources.<\/p>\n\n\n\n<p>Approach: First, the AR collaborative drawing \u009d functionality will be implemented using touchscreens with the AR-eProc system developed at NASA Johnson Space Flight Center. Next, gestures appropriate for hands-free drawing \u009d will be developed. These gestures will be integrated into AR-eProc. Sample instruction modules, in increasing sophistication: without AR, with static AR images, with touchscreen AR collaboration, and with radar-based AR gesture collaboration; will be tested with volunteers. The system\u2019s impact on user comfort, efficiency, and perceived utility in job environments will be assessed. Iterative, agile approaches will be used to modify software development as assessments reveal areas of needed improvement.<\/p>\n\n\n\n<p>Evaluation of AR platforms will use task completion time, the subjective workload assessment tool NASA TLX, and subjective surveys of test subjects to link AR interactivity with changing levels of productivity. Feasibility of radar gesture recognition will be evaluated as recognition accuracy as the percentage of recognized gestures from the number of properly executed gestures during testing.<\/p>\n\n\n\n<p>Needed Resources: The AR-eProc developed at NASA Johnson, especially the hardware headset, is needed. Additional Microsoft Hololens sets will be available for sale in 2016 for the price of $3000 each. For radar-based gesture detection, a development kit from Google ATAP\u2019s Project Soli would provide the sensor and application programming interface, allowing for the rapid creation of gesture recognition patterns. Otherwise, microwave Doppler sensors are available commercially off-the-shelf. Doshisha University in Japan has demonstrated gesture recognition using the commercial NJR4211J Microwave Doppler sensor.<\/p>\n\n\n\n<p><a href=\"https:\/\/www.nasa.gov\/directorates\/spacetech\/strg\/nstrf-2016\"><strong>Back to NSTRF 2016 Full List<\/strong><\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hiroshi Furuya Columbia University Augmented Reality (AR), where 3D and 2D images are overlaid on a user\u2019s natural field of view like a heads up display, is being developed for use in manned space applications. In the past 2 years, NASA, through AR-eProc, and the European Space Agency, through MARSOP, have demonstrated the potential of […]<\/p>\n","protected":false},"author":66,"featured_media":63425,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"nasa_hds_core_meta_l10n":"en-US","nasa_hds_core_meta_article_editor":"","nasa_hds_core_meta_article_contacts":[],"nasa_hds_core_meta_article_topic":"","nasa_hds_core_meta_linked_locations":[],"nasa_hds_core_meta_article_related_articles":[],"hds_migrate_meta_pr_release_id":"","hds_migrate_meta_pr_release_type":"","nasa_hds_core_meta_location_x":"","nasa_hds_core_meta_location_y":"","nasa_hds_core_meta_gallery_images":[],"footnotes":""},"categories":[3337],"news-tags":[],"internal-terms":[9529,12750],"coauthors":[],"parsely":{"version":"1.1.0","meta":{"@context":"https:\/\/schema.org","@type":"NewsArticle","headline":"Collaborative Augmented Reality with Hands-Free Gesture Control for Remote Astronaut Training and Mentoring","url":"https:\/\/www.nasa.gov\/directorates\/stmd\/space-tech-research-grants\/collaborative-augmented-reality-with-hands-free-gesture-control-for-remote-astronaut-training-and-mentoring\/","mainEntityOfPage":{"@type":"WebPage","@id":"https:\/\/www.nasa.gov\/directorates\/stmd\/space-tech-research-grants\/collaborative-augmented-reality-with-hands-free-gesture-control-for-remote-astronaut-training-and-mentoring\/"},"thumbnailUrl":"https:\/\/www.nasa.gov\/wp-content\/uploads\/2016\/07\/furuya_hiroshi.jpg?w=150&h=150&crop=1","image":{"@type":"ImageObject","url":"https:\/\/www.nasa.gov\/wp-content\/uploads\/2016\/07\/furuya_hiroshi.jpg"},"articleSection":"NASA Directorates","author":[{"@type":"Person","name":"Loura Hall"}],"creator":["Loura Hall"],"publisher":{"@type":"Organization","name":"NASA","logo":"https:\/\/www.nasa.gov\/wp-content\/themes\/nasa\/assets\/images\/nasa-logo.svg"},"keywords":["NASA Directorates","Space Technology Mission Directorate","Space Technology Research Grants"],"dateCreated":"2016-07-19T19:16:00Z","datePublished":"2016-07-19T19:16:00Z","dateModified":"2023-07-26T19:12:59Z"},"rendered":"<meta name=\"parsely-title\" content=\"Collaborative Augmented Reality with Hands-Free Gesture Control for Remote Astronaut Training and Mentoring\" \/>\n<meta name=\"parsely-link\" content=\"https:\/\/www.nasa.gov\/directorates\/stmd\/space-tech-research-grants\/collaborative-augmented-reality-with-hands-free-gesture-control-for-remote-astronaut-training-and-mentoring\/\" \/>\n<meta name=\"parsely-type\" content=\"post\" \/>\n<meta name=\"parsely-image-url\" content=\"https:\/\/www.nasa.gov\/wp-content\/uploads\/2016\/07\/furuya_hiroshi.jpg?w=150&h=150&crop=1\" \/>\n<meta name=\"parsely-pub-date\" content=\"2016-07-19T19:16:00Z\" \/>\n<meta name=\"parsely-section\" content=\"NASA Directorates\" \/>\n<meta name=\"parsely-tags\" content=\"NASA Directorates,Space Technology Mission Directorate,Space Technology Research Grants\" \/>\n<meta name=\"parsely-author\" content=\"Loura Hall\" \/>","tracker_url":"https:\/\/cdn.parsely.com\/keys\/nasa.gov\/p.js"},"read_time":3,"time_ago":"8 years","featured_image":{"id":63425,"file":"https:\/\/www.nasa.gov\/wp-content\/uploads\/2016\/07\/furuya_hiroshi.jpg","title":"furuya_hiroshi","alt":"Hiroshi Furuya","caption":""},"_links":{"self":[{"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/posts\/61357"}],"collection":[{"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/users\/66"}],"replies":[{"embeddable":true,"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/comments?post=61357"}],"version-history":[{"count":1,"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/posts\/61357\/revisions"}],"predecessor-version":[{"id":463498,"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/posts\/61357\/revisions\/463498"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/media\/63425"}],"wp:attachment":[{"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/media?parent=61357"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/categories?post=61357"},{"taxonomy":"news-tags","embeddable":true,"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/news-tags?post=61357"},{"taxonomy":"internal-terms","embeddable":true,"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/internal-terms?post=61357"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.nasa.gov\/wp-json\/wp\/v2\/coauthors?post=61357"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}