Source: https://patents.google.com/patent/US9509981
Timestamp: 2018-02-19 17:08:20
Document Index: 479084064

Matched Legal Cases: ['Application No. 200980139375', 'Application No. 11747870', 'Application No. 11747870', 'Application No. 200980139375', 'Application No. 200980139375', 'Application No. 200980139375', 'Application No. 201180010523', 'Application No. 2011', 'Application No. 2012', 'Application No. 200980139375', 'Application No. 12752325', 'Application No. 2011', 'Application No. 2011', 'Application No. 2012', 'Application No. 2014', 'Application No. 2014', 'Application No. 200980139375', 'Application No. 2011', 'Application No. 2011', 'Application No. 200980139375', 'Application No. 11747870', 'Application No. 200980139375', 'Application No. 11747870', 'Application No. 12752325', 'Application No. 201210037666', 'Application No. 11747870']

US9509981B2 - Projectors and depth cameras for deviceless augmented reality and interaction - Google Patents
US9509981B2
US9509981B2 US14281885 US201414281885A US9509981B2 US 9509981 B2 US9509981 B2 US 9509981B2 US 14281885 US14281885 US 14281885 US 201414281885 A US201414281885 A US 201414281885A US 9509981 B2 US9509981 B2 US 9509981B2
US14281885
US20140253692A1 (en )
This application is a continuation of, and claims benefit from, commonly assigned, co-pending U.S. patent application Ser. No. 12/819,230, with inventors Andrew David Wilson et al., filed Jun. 21, 2010, entitled “Projectors and Depth Cameras for Deviceless Augmented Reality and Interaction,” which further claims the benefit of U.S. Provisional Patent application Ser. No. 61/307,422 entitled “PROJECTOR AND DEPTH CAMERA SYSTEMS FOR DEVICELESS AUGMENTED REALITY AND INTERACTION,” filed Feb. 23, 2010, the entireties of both of which are incorporated by reference herein.
The capability of the disclosed architecture to track and reason about the interactions throughout the space (e.g., room) facilitates the connection of individual interactive surfaces into a seamless monolithic interactive space. A user can transition objects between interactive surfaces “through-body” by simply touching the object and then touching the desired location. The system infers that both contacts belong to the same person, establishing a connection between surfaces. For example, when the user touches an image on the table and then also touches the wall, the image is transitioned to the wall. In other words, through-body transitions are accomplished by simultaneously touching two surfaces.
In addition to making connections through-body, the user can literally drag an object off an interactive surface and pick it up by hand. The system does not actually track the user's hands (or any other body part), but rather gives each object a physics-like behavior. Picking up objects from a table, for example, can be accomplished by swiping the object into a hand. Following the pick-up of the object, an iconic representation of the object can be presented in association with in the hand.
an arrangement of projectors that impose projections relative to the physical space, with the projections interrelated such that objects in at least two projections are cross-referenced to establish connectivity in the physical space,
at least a portion of the connectivity computed by constructing tables that index connecting components from one view to another view;
a mapping component that maps portions of the sensed information to the projections to create mapping information; and
a modeling component that creates a multi-dimensional model based on the mapping information, the multi-dimensional model describing a relationship between output of the cameras and the projectors for virtual interaction between a user and an object in the physical space.
2. The system of claim 1, wherein the mapping component maps portions of the sensed information to the projections to create a 3D physical coordinate system.
3. The system of claim 1, further comprising a control component that applies the sensed information, which is 3D sensed information, to the projections, which are 2D images.
5. The system of claim 1, further comprising a control component that facilitates projection of a spatial widget on a surface in the physical space, and wherein interactive manipulation of the spatial widget is accomplished using a predetermined movement in the physical space.
6. The system of claim 1, wherein the sensed information of the depth-sensing cameras is utilized to construct a 3D mesh, wherein the system comprises a control component that processes the 3D mesh to simulate positioning of a virtual object relative to a user in the physical space.
8. The system of claim 1, further comprising a control component that processes the sensed information and the projections to map a virtual interactive space to the physical space to emulate an interactive display for interaction with the objects and the surfaces.
9. The system of claim 8, wherein the control component that processes the sensed information to infer depth of the objects in the physical space.
10. A multi-dimensional modeling method, comprising:
orienting an arrangement of depth-sensing cameras that sense objects in a physical space and output sensed information;
orienting an arrangement of projectors that impose projections relative to the physical space, with the projections interrelated such that objects in at least two projections are cross-referenced to establish connectivity in the physical space,
mapping portions of the sensed information to the projections to create mapping information; and
generating a multi-dimensional model based on the mapping information, the multi-dimensional model describing a relationship between output of the cameras and the projectors for virtual interaction between a user and an object in the physical space.
11. The method of claim 10, further comprising mapping portions of the sensed information to the projections to create a 3D physical coordinate system.
12. The method of claim 10, further comprising applying the sensed information, which is 3D sensed information, to the projections, which are 2D images.
13. The method of claim 10, wherein the cameras and the projectors are registered to the physical space.
14. The method of claim 10, further comprising facilitating projection of a spatial widget on a surface in the physical space, and wherein interactive manipulation of the spatial widget is accomplished using a predetermined movement in the physical space.
orienting an arrangement of depth-sensing cameras to sense objects within a physical space;
orienting an arrangement of projectors to project images in the physical space via projections, with the projections interrelated such that objects in at least two projections are cross-referenced to establish connectivity in the physical space, at least a portion of the connectivity computed by constructing tables that index connecting components from one view to another view; and
creating a multi-dimensional model that describes a relationship between output of the cameras and the projectors for virtual interaction between a user and an object in the space.
US14281885 2010-02-23 2014-05-19 Projectors and depth cameras for deviceless augmented reality and interaction Active 2031-07-16 US9509981B2 (en)
US12819230 Continuation US8730309B2 (en) 2010-02-23 2010-06-21 Projectors and depth cameras for deviceless augmented reality and interaction
US20140253692A1 true US20140253692A1 (en) 2014-09-11
US9509981B2 true US9509981B2 (en) 2016-11-29
US9704267B2 (en) * 2015-06-15 2017-07-11 Electronics And Telecommunications Research Institute Interactive content control apparatus and method
CN105531756B (en) * 2013-09-17 2017-11-24 索尼公司 The information processing apparatus, information processing method and a recording medium
JPH07129319A (en) 1993-11-01 1995-05-19 Hitachi Ltd Pen input type information processor
JPH07168949A (en) 1993-04-30 1995-07-04 Rank Xerox Ltd Interactive copying device
US5552839A (en) 1992-10-26 1996-09-03 Nsm Aktiengesellschaft Projection system for entertainment and game apparatus
JPH11134327A (en) 1997-08-26 1999-05-21 Sharp Corp Electronic memo processor and method therefor and computer readable recording medium for recording electronic memo processing program
US20010035845A1 (en) 1995-11-28 2001-11-01 Zwern Arthur L. Portable display and method for controlling same with speech
US20010048429A1 (en) 1998-08-13 2001-12-06 Reynold Liao Computer system having a configurable touchpad-mouse button combination
CN1378741A (en) 1999-08-10 2002-11-06 彼得·麦克达菲·怀特 Communication Systems
US20030034976A1 (en) 2001-08-14 2003-02-20 Ramesh Raskar System and method for registering multiple images with three-dimensional objects
US20030042401A1 (en) 2000-04-25 2003-03-06 Hansjorg Gartner Combined stereovision, color 3D digitizing and motion capture system
US20030146898A1 (en) 2002-02-07 2003-08-07 Gifu University Touch sense interface and method for controlling touch sense interface
US20040257540A1 (en) 2003-04-16 2004-12-23 Sebastien Roy Single or multi-projector for arbitrary surfaces without calibration nor reconstruction
US20050017739A1 (en) 2003-04-23 2005-01-27 Hamren Steven L. Method and apparatus for processing semiconductor devices in a singulated form
JP2005031747A (en) 2003-07-07 2005-02-03 Fujitsu Component Ltd Pointing device control method and apparatus
WO2005017739A1 (en) 2003-08-19 2005-02-24 Koninklijke Philips Electronics N.V. A visual content signal display apparatus and a method of displaying a visual content signal therefor
US20050099603A1 (en) 2002-03-15 2005-05-12 British Broadcasting Corporation Virtual studio system
US20050135670A1 (en) 2003-12-17 2005-06-23 Janakiraman Vaidyanathan CAD modeling system and method
US20050185150A1 (en) 2004-02-20 2005-08-25 Turner James A. Image display system and method for head-supported viewing system
US20060001645A1 (en) 2004-06-30 2006-01-05 Microsoft Corporation Using a physical object to control an attribute of an interactive display application
US20070013716A1 (en) 2002-08-23 2007-01-18 International Business Machines Corporation Method and system for a user-following interface
US20070055782A1 (en) 2004-03-29 2007-03-08 William Wright System and method for applying link analysis tools for visualizing connected temporal and spatial information on a user inferface
US20070126938A1 (en) 2005-12-05 2007-06-07 Kar-Han Tan Immersive surround visual fields
US20070134456A1 (en) 1992-06-05 2007-06-14 Fritschen Thomas M Multi Directional Fibers In A Shell
JP2007299384A (en) 2006-05-03 2007-11-15 Mitsubishi Electric Research Laboratories Inc Method and system for emulating mouse on multi-touch sensitive screen executed by computer
US20070299338A1 (en) 2004-10-14 2007-12-27 Stevick Glen R Method and apparatus for dynamic space-time imaging system
US20080095468A1 (en) 2004-08-30 2008-04-24 Bauhaus-Universitaet Weimar Method And Device For Representing A Digital Image On A Surface Which Is Non-Trivial In Terms Of Its Geometry And Photometry
US20080180402A1 (en) 2007-01-25 2008-07-31 Samsung Electronics Co., Ltd. Apparatus and method for improvement of usability of touch screen
US20080229194A1 (en) 2007-03-14 2008-09-18 Microsoft Corporation Virtual features of physical items
US20080246781A1 (en) 2007-03-15 2008-10-09 Scalable Display Technologies, Inc. System and method for providing improved display quality by display adjustment and image processing using optical feedback
US20080263458A1 (en) 2007-04-20 2008-10-23 Utbk, Inc. Methods and Systems to Facilitate Real Time Communications in Virtual Reality
US20080316201A1 (en) 2004-10-25 2008-12-25 Columbia Unversity Systems and Methods for Displaying Three-Dimensional Images
US20090037841A1 (en) 2000-04-24 2009-02-05 Bell Blaine A System And Method For Dynamic Space Management Of A Display Space
US20090059041A1 (en) 2007-08-27 2009-03-05 Sung Jin Kwon Method of correcting image distortion and apparatus for processing image using the method
US20090079942A1 (en) 2004-10-30 2009-03-26 Sung-Ha Lee Automatic Vision Display Apparatus Using Pursuit of Flying Path of Flying Screen Unit
US20090091581A1 (en) 2007-10-05 2009-04-09 Artec Ventures Combined Object Capturing System and Display Device and Associated Method
US20090167966A1 (en) 2008-01-02 2009-07-02 Samsung Electronics Co., Ltd. See-through display apparatus and method thereof
US20090217187A1 (en) 2005-02-12 2009-08-27 Next Device Ltd User Interfaces
WO2010019802A1 (en) 2008-08-15 2010-02-18 Gesturetek, Inc. Enhanced multi-touch detection
US20100063681A1 (en) 2006-11-27 2010-03-11 Carl Zeiss Microimaging Gmbh Method and arrangement for the steering of a vehicle
US20100073366A1 (en) 2008-09-24 2010-03-25 Canon Kabushiki Kaisha Model generation apparatus and method
US20100103386A1 (en) 2008-10-29 2010-04-29 Seiko Epson Corporation Projector and projector control method
US20100201878A1 (en) 2006-03-31 2010-08-12 Koninklijke Philips Electronics N.V. Adaptive content rendering based on additional frames of content
US20100238107A1 (en) 2009-03-19 2010-09-23 Yoshihito Ohki Information Processing Apparatus, Information Processing Method, and Program
US20110058709A1 (en) 2009-01-30 2011-03-10 Microsoft Corporation Visual target tracking using model fitting and exemplar
US20110216948A1 (en) 2010-03-04 2011-09-08 Flashscan3D, Llc System and method for three-dimensional biometric data feature detection and recognition
EP2378310A1 (en) 2010-04-15 2011-10-19 Cedes Safety & Automation AG Time of flight camera unit and optical surveillance system
US20110282140A1 (en) 2010-05-14 2011-11-17 Intuitive Surgical Operations, Inc. Method and system of hand segmentation and overlay using depth data
US20110304691A1 (en) 2009-02-17 2011-12-15 Koninklijke Philips Electronics N.V. Combining 3d image and graphical data
US20110310226A1 (en) 2010-06-16 2011-12-22 Microsoft Corporation Use of wavefront coding to create a depth image
KR20120018915A (en) 2010-08-24 2012-03-06 삼성전자주식회사 Apparatus and method for generating depth image that have same viewpoint and same resolution with color image
US20120105585A1 (en) 2010-11-03 2012-05-03 Microsoft Corporation In-home depth camera calibration
JP2012510673A (en) 2008-12-03 2012-05-10 アルカテル−ルーセント Mobile device for augmented reality applications
US20120140038A1 (en) 2010-12-01 2012-06-07 Qualcomm Incorporated Zero disparity plane for feedback-based three-dimensional video
US20120154619A1 (en) 2010-12-17 2012-06-21 Qualcomm Incorporated Augmented reality processing based on eye capture in handheld device
US20120194516A1 (en) 2011-01-31 2012-08-02 Microsoft Corporation Three-Dimensional Environment Reconstruction
US20120223909A1 (en) 2011-03-02 2012-09-06 Smart Technologies Ulc 3d interactive input system and method
US20120268570A1 (en) 2009-12-24 2012-10-25 Trumbull Ventures Llc Method and apparatus for photographing and projecting moving images in three dimensions
US20120274745A1 (en) 2011-04-29 2012-11-01 Austin Russell Three-dimensional imager and projection device
US20120315965A1 (en) 2011-06-08 2012-12-13 Microsoft Corporation Locational Node Device
US20130002815A1 (en) 2011-07-01 2013-01-03 Disney Enterprises, Inc. 3d drawing system for providing a real time, personalized, and immersive artistic experience
US20130176539A1 (en) 2010-09-10 2013-07-11 Lemoptix Sa Method and device for projecting a 3-d viewable image
US20140051510A1 (en) 2011-03-02 2014-02-20 Hrvoje Benko Immersive display with peripheral illusions
WO2014028276A1 (en) 2012-08-14 2014-02-20 Microsoft Corporation Wide angle depth detection
US6369805B1 (en) 1993-11-01 2002-04-09 Hitachi, Ltd. Information processing system
US20100182416A1 (en) 2006-05-24 2010-07-22 Smart Technologies Ulc Method and apparatus for inhibiting a subject's eyes from being exposed to projected light
US20130229353A1 (en) 2008-09-30 2013-09-05 Microsoft Corporation Using Physical Objects in Conjunction with an Interactive Surface
WO2011106201A2 (en) 2010-02-23 2011-09-01 Microsoft Corporation Projectors and depth cameras for deviceless augmented reality and interaction
"Applicant-Initiated Interview Summary in U.S. Appl. No. 12/241,281", Mailed Date: Mar. 22, 2012, 3 Pages.
"Applicant-Initiated Interview Summary in U.S. Appl. No. 12/241,281", Mailed Date: Nov. 9, 2011, 3 Pages.
"Applicant-Initiated Interview Summary in U.S. Appl. No. 12/241,281", Mailed Date: Sep. 20, 2012, 3 Pages.
"Auto Yoke the First Automated Lighting Solution Designed for Theatrical Use", Retrieved from <<http://web.archive.org/web/20120309060116/http://www.citytheatrical.com/rcyoke.htm>>, Retrieved on: Mar. 9, 2012, 2 Pages.
"Decision on Re-examination Issued in Chinese Patent Application No. 200980139375.3", Mailed Date: Jan. 28, 2015, 11 Pages.
"Decision to refuse a European Patent application," Received in European Patent Application No. 11747870.1, Mailed Date: Jun. 3, 2015, 4 Pages.
"Dynamical simulation", Retrieved on <<https://en.wikipedia.org/wiki/Dynamical-simulation>>, Wikipedia Entry, Retrieved on: Mar. 9, 2011, 4 Pages.
"Examination Report Received in European Patent Application No. 11747870.1", Mailed Date: May 23, 2014, 7 Pages.
"Final Office Action Issued in Chinese Patent Application No. 200980139375.3", Mailed Date: Jan. 8, 2014, 10 Pages.
"Final Office Action Issued in U.S. Appl. No. 12/241,281", Mailed Date: Jan. 6, 2012, 27 Pages.
"Final Office Action Issued in U.S. Appl. No. 12/819,230", Mailed Date: May 9, 2013, 20 Pages.
"Final Office Action Issued in U.S. Appl. No. 13/039,179", Mailed Date: Nov. 5, 2014, 25 Pages.
"Final Office Action Issued in U.S. Appl. No. 13/074,041", Mailed Date: Jan. 8, 2014, 23 Pages.
"Final Office Action Issued in U.S. Appl. No. 13/074,041", Mailed Date: Oct. 9, 2014, 25 Pages.
"Final Office Action Issued in U.S. Appl. No. 13/084,786", Mailed Date: Jul. 17, 2013, 27 Pages.
"Final Office Action Issued in U.S. Appl. No. 13/155,422", Mailed Date: Aug. 16, 2013, 22 Pages.
"First Office Action Issued in Chinese Patent Application No. 200980139375.3", Mailed Date: Dec. 18, 2012, 14 Pages.
"Great Films Fill Rooms", Retrieved from <<http://greatfilmsfillrooms.com/en/>>, Retrieved on: Feb. 19, 2013, 1 Page.
"International Preliminary Report on Patentability Issued in PCT Application No. PCT/US2009/054607", Mailed Date: Apr. 5, 2011, 5 Pages.
"International Preliminary Report on Patentability Issued in PCT Patent Application No. PCT/US2011/024925", Mailed Date: Sep. 7, 2012, 6 Pages.
"International Preliminary Report on Patentability Issued in PCT Patent Application No. PCT/US2012/26823", Mailed Date Sep. 12, 2013, 1 Page.
"International Search Report & Written Opinion Issued in PCT Application No. PCT/US2012/24786", Mailed Date: Oct. 18, 2012, 8 Pages.
"International Search Report and Written Opinion for PCT Application No. PCT/ US2012/026823", Mailed Date: Sep. 28, 2012, Filed Date: Feb. 27, 2012, 6 Pages.
"International Search Report and Written Opinion Issued in PCT Application No. PCT/US2009/054607", Mailed Date: Apr. 6, 2010, 11 Pages.
"International Search Report Issued in PCT Application No. PCT/US2012/26823", Mailed Date: Sep. 28, 2012, 6 Pages.
"Liquid Crystal Shutter Glasses", Retrieved from <<http://en.wikipedia.org/wiki/Active-shutter-glasses>>, Retrieved on: Apr. 8, 2011, 6 Pages.
"Microsoft Office Labs", Retrieved from <<http://www.microsoft.com/office/labs/index.html>>, Retrieved on: Feb. 19, 2013, 7 Pages.
"Microsoft Surface", Retrieved from <<http://www.surface.com>>, Retrieved on: Jan. 7, 2009, 1 Page.
"Newton Game Dynamics", Retrieved from <<http://web.archive.org/web/20130222075951/http://newtondynamics.com/forum/newton.php>>, Retrieved on: Feb. 22, 2013, 2 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/039,179", Mailed Date: Jul. 2, 2014, 24 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/039,179", Mailed Date: Nov. 29, 2013, 22 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/074,041", Mailed Date: Apr. 2, 2015, 19 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/074,041", Mailed Date: Aug. 23, 2013, 20 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/074,041", Mailed Date: Feb. 1, 2013, 22 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/074,041", Mailed Date: Jun. 23, 2014, 21 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/084,786", Mailed Date: Jul. 8, 2015, 29 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/084,786", Mailed Date: Mar. 12, 2013, 24 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/891,116", Mailed Date Jan. 14, 2015, 13 Pages.
"Non Final Office Action Issued in U.S. Appl. No. 13/891,116", Mailed Date: Oct. 28, 2015, 7 Pages.
"Non-final Office Action Issued in U.S. Appl. No. 12/241,281", Mailed Date: Aug. 25, 2011, 26 Pages.
"Non-final Office Action Issued in U.S. Appl. No. 12/241,281", Mailed Date: Jul. 20, 2012, 21 Pages.
"Non-Final Office Action Issued in U.S. Appl. No. 12/819,230", Mailed Date: Dec. 24, 2012, 18 Pages.
"Non-final Office Action Issued in U.S. Appl. No. 13/155,422", Mailed Date: Apr. 11, 2014, 9 Pages.
"Non-final Office Action Issued in U.S. Appl. No. 13/155,422", Mailed Date: Dec. 20, 2012, 10 Pages.
"Non-final Office Action Issued in U.S. Appl. No. 13/155,422", Mailed Date: Mar. 26, 2015, 19 Pages.
"Non-final Office Action Issued in U.S. Appl. No. 13/851,797", Mailed Date: Apr. 10, 2014, 22 Pages.
"Non-final Office Action Issued in U.S. Appl. No. 14/037,986", Mailed Date: Apr. 27, 2015, 21 Pages.
"Notice of Allowance Issued in Chinese Patent Application No. 200980139375.3", Mailed Date: Mar. 2, 2015, 4 Pages.
"Notice of Allowance Issued in Chinese Patent Application No. 201180010523.9", Mailed Date: Jan. 21, 2014, 6 Pages.
"Notice of Allowance Issued in Japanese Patent Application No. 2011-530082", Mailed Date: Aug. 5, 2014, 3 Pages.
"Notice of Allowance Issued in Japenese Patent Application No. 2012-555036", Mailed Date: Apr. 21, 2015, 3 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 12/241,281", Mailed Date: Dec. 24, 2012, 10 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 12/819,230", Mailed Date: Jan. 8, 2014, 9 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 13/074,041", Mailed Date: Sep. 16, 2015, 5 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 13/155,422", Mailed Date: Aug. 15, 2014, 8 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 13/155,422", Mailed Date: Sep. 15, 2015, 11 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 13/851,797", Mailed Date: Aug. 21, 2014, 10 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 13/851,797", Mailed Date: Dec. 5, 2014, 13 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 13/851,797", Mailed Date: Jul. 6, 2015, 10 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 13/851,797", Mailed Date: Mar. 24, 2015, 10 Pages.
"Notice of Allowance Issued in U.S. Appl. No. 13/851,797", Mailed Date: Nov. 5, 2015, 10 Pages.
"Office Action Issued in Chinese Patent Application No. 200980139375.3", Mailed Date: Aug. 7, 2014, 5 Pages.
"Office Action Issued in European Patent Application No. 12752325.6", Mailed Date: Aug. 6, 2014, 8 Pages.
"Office Action Issued in Japanese Patent Application No. 2011-530082", Mailed Date: Mar. 11, 2014, 8 Pages.
"Office Action Issued in Japanese Patent Application No. 2011-530082", Mailed Date: Oct. 1, 2013, 8 Pages.
"Office Action Issued in Japanese Patent Application No. 2012-555036", Mailed Date: Dec. 4, 2014, 4 Pages.
"Office Action Issued in Japanese Patent Application No. 2014-142550", Mailed Date: Aug. 25, 2015, 8 Pages.
"Office Action Issued in Japanese Patent Application No. 2014-142551", Mailed Date: Aug. 18, 2015, 8 Pages.
"Optimus Maximus Keyboard", Retrieved from <<http://web.archive.org/web/20090105063004/http://www.artlebedev.com/everything/optimus/>>, Jan. 5, 2009, 5 Pages.
"Physics engine", Retrieved on <<https://en.wikipedia.org/wiki/Physics-engine>>, Wikipedia entry, Retrieved on: Mar. 9, 2011, 7 Pages.
"PMD[vision] CamBoard nano reference design," PMD Technologies, , Access date: Aug. 10, 2012, 3 pages.
"PMD[vision] CamBoard nano reference design," PMD Technologies, <http://www.pmdtec.com/products-services/pmdvisionr-cameras/pmdvisionr-camboard-nano/>, Access date: Aug. 10, 2012, 3 pages.
"Response to Final Office Action mailed Date Aug. 16, 2013 in U.S. Appl. No. 13/155,422", filed Nov. 14, 2013, 14 Pages.
"Response to Final Office Action mailed Date Jan. 6, 2012 in U.S. Appl. No. 12/241,281", filed May 2, 2012, 11 Pages.
"Response to Final Office Action Mailed Date Jan. 8, 2014 in U.S. Appl. No. 13/074,041", filed Apr. 30, 2014, 19 Pages.
"Response to Final Office Action mailed Date Jul. 17, 2013 in U.S. Appl. No. 13/084,786", filed Oct. 17, 2013, 16 Pages.
"Response to Non-Final Office Action Mailed Date Apr. 10, 2014 in U.S. Appl. No. 13/851,797", filed Jun. 16, 2014, 14 Pages.
"Response to Non-Final Office Action mailed Date Apr. 11, 2014 in U.S. Appl. No. 13/155,422", filed Jul. 11, 2014, 11 Pages.
"Response to Non-Final Office Action Mailed Date Aug. 23, 2013 in U.S. Appl. No. 13/074,041", filed Dec. 3, 2013, 19 Pages.
"Response to Non-Final Office Action mailed Date Aug. 25, 2011 in U.S. Appl. No. 12/241,281", filed Nov. 2, 2011, 13 Pages.
"Response to Non-Final Office Action mailed Date Dec. 20, 2012 in U.S. Appl. No. 13/155,422", filed Apr. 22, 2013, 12 Pages.
"Response to Non-Final Office Action Mailed Date Feb. 1, 2013 in U.S. Appl. No. 13/074,041", filed May 29, 2013, 15 Pages.
"Response to Non-Final Office Action mailed Date Jul. 20, 2012 in U.S. Appl. No. 12/241,281", filed Oct. 22, 2012, 13 Pages.
"Response to Non-Final Office Action Mailed Date Jun. 23, 2014 in U.S. Appl. No. 13/074,041", filed Sep. 23, 2014, 16 Pages.
"Response to Non-Final Office Action mailed Date Mar. 12, 2013 in U.S. Appl. No. 13/084,786", filed Jun. 12, 2013, 13 Pages.
"Response to Non-Final Office Action mailed Date Mar. 26, 2015 in U.S. Appl. No. 13/155,422", filed Jul. 22, 2015, 12 Pages.
"Response to Non-Final Office Action Mailed Date Oct. 9, 2014 in U.S. Appl. No. 13/074,041", filed Jan. 9, 2015, 15 Pages.
"Response to Notice of Re-examination Mailed Aug. 7, 2014 in Chinese Patent Application No. 200980139375.3", Filed Date: Oct. 9, 2014, 11 Pages.
"Response to Office Action Dated Mar. 11, 2014 for Japanese Application No. 2011-530082", Filed Date: Jul. 10, 2014, 6 Pages.
"Response to Office Action Dated Oct. 1, 2013 for Japanese Application No. 2011-530082", Filed Date: Dec. 26, 2013, 11 Pages.
"Response to Office Action mailed Date Apr. 2, 2015 in U.S. Appl. No. 13/074,041", filed Aug. 21, 2015, 14 Pages.
"Response to Restriction Requirement Mailed Date Jun. 21, 2011 in U.S. Appl. No. 12/241,281", filed Jul. 20, 2011, 7 Pages.
"Response to the Second Office Action Dated Aug. 7, 2013 for Chinese Patent Application No. 200980139375.3", Filed Date: Aug. 30, 2013, 15 Pages.
"Restriction Requirement Issued in U.S. Appl. No. 12/241,281", Mailed Date: Jun. 21, 2011, 7 Pages.
"Search Report Received in European Patent Application No. 11747870.1", Mailed Date: May 13, 2014, 3 Pages.
"Second Office Action Issued in Chinese Patent Application No. 200980139375.3", Mailed Date: Aug. 7, 2013, 12 Pages.
"Second Written Opinion Issued in PCT Application No. PCT/US2012/26823", Mailed Date Sep. 28, 2012, 5 Pages.
"Sketching User Experiences", Retrieved from <<http://www.billbuxton.com/>>, Retrieved on: Jan. 7, 2009, 6 Pages.
"Steerable Augmented Reality with the Beamatron", Retrieved from <<http://research.microsoft.com/en-us/um/people/awilson/publications/WilsonUIST2012/WilsonUIST2012.html>>, Retrieved on: Oct. 2012, 1 Page.
"Stereoscopy", Retrieved from << http://https://web.archive.org/web/20110409235522/http://en.wikipedia.org/wiki/Stereoscopy>>, Retrieved on: Apr. 8, 2011, 21 Pages.
"Stereoscopy", Retrieved from <<http://en.wikipedia.org/wiki/Stereoscopy#Complementary-color-anaglyphs>>, Retrieved on: Apr. 8, 2011, 34 Pages.
"Structured Light 3D Scanning", Retrieved from <<https://sites.google.com/site/structuredlight/techniques>>, Retrieved on: Nov. 26, 2010, 3 Pages.
"Summons to Oral Proceedings" Received in European Patent Application No. 11747870.1, Mailed Date: Nov. 10, 2014, 9 Pages.
"Supplementary Search Report for European Patent Application No. 12752325.6", Mailed Date: Jul. 25, 2014, 4 Pages.
"Voluntary Amendment in Chinese Patent Application No. 201210037666.5", Filed Date: May 29, 2014, 13 Pages.
Agarawala, et al., "Keepin' It Real: Pushing the Desktop Metaphor with Physics, Piles and the Pen", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 2006, pp. 1283-1292.
Allen, Jonas, "I-Cocoon: The Next Wii?" Retrieved from <<http://www.dailygame.net/electronics/i-cocoon-the-next-wii>>, Sep. 11, 2008, 4 Pages.
Allen, Jonas, "I-Cocoon: The Next Wii?" Retrieved from >, Sep. 11, 2008, 4 Pages.
Anderson, et al., "Tangible Interaction+ Graphical Interpretation: A New Approach to 3D Modeling", In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, Apr. 2000, 14 Pages.
Ashdown, et al., "Steerable Projector Calibration", In IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, Published on Jun. 2005, 8 Pages.
Ates, et al., "Immersive Simulation of Visual Impairments Using a Wearable See-through Display", In Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, Jan. 2015, 4 Pages.
Baribeau, et al., "Color Reflectance Modeling Using a Polychromatic Laser Range Sensor", In Proceedings on IEEE Transactions of Pattern Analysis and Machine Intelligence, vol. 14, Issue 2, Feb. 1992, pp. 263-269.
Bell, et al., "Dynamic Space Management for User Interfaces", In ACM Symposium on User Interface Software and Technology, Nov. 2000, pp. 239-248.
Benko, et al., "Dyadic Projected Spatial Augmented Reality", In Proceedings of the 27th annual ACM Symposium on User Interface Software and Technology, Oct. 2014, 11 Pages.
Benko, et al., "Mirage Table: Freehand Interaction on a Projected Augmented Reality Tabletop", In CHI '12 Proceedings of the SIGCHI Conference on Human Factors in Computing System, May 5, 2012, 10 Pages.
Benko, et al., "Multi-Point Interactions with Immersive Omnidirectional Visualizations in a Dome", In ACM International Conference on Interactive Tabletops and Surfaces, Nov. 7, 2010, 10 Pages.
Billinghurst, et al., "Shared Space-An Augmented Reality Approach for Computer Supported Collaborative Work", Virtual Reality vol. 3, No. 1, Mar. 1998, 12 Pages.
Billinghurst, et al., "Tangible Augmented Reality", In ACM Special Interest Group on Computer Graphics and Interactive Techniques, Jan. 2008, pp. 1-10.
Bimber, et al., "Spatial Augmented Reality: Merging Real and Virtual Worlds", AK Peters, Ltd. / CRC Press, Aug. 8, 2005, 393 Pages.
Bishop, Todd, "Q&A: Microsoft LightSpace, One Step Closer to Star Trek Holodeck", Retrieved from <<http://www.bizjournals.com/seattle/blog/techflash/2010/10/qa-microsoft-lightspace-one-step-closer-to-star-trek-holodeck.html>>, Oct. 7, 2010, 4 Pages.
Blackburn, et al., "Incremental 3D Model Generation using Depth Cameras", Retrieved at <<http://www.ifp.illinois.edu/˜kubacki1/Doc/ECE549FinalProjectReport.pdf>>, Retrieved Date: Mar. 29, 2012, pp. 1-8.
Blackburn, et al., "Incremental 3D Model Generation using Depth Cameras", Retrieved at >, Retrieved Date: Mar. 29, 2012, pp. 1-8.
Bolt, et al., "Put-That-There": Voice and Gesture at the Graphics Interface, In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques, Jul. 14, 1980, 9 Pages.
Borkowski, et al., "Spatial Control of Interactive Surfaces in an Augmented Environment", In Proceedings of the International Conference on Engineering Human Computer Interaction and Interactive Systems, Jul. 11, 2004, 16 Pages.
Boverie, et al., "Comparison of Structured Light and Stereovision Sensors for New Airbag Generations", In Journal of Control Engineering Practice, vol. 11, No. 12, Dec. 2003, pp. 1413-1421.
Brignull, et al., "The Introduction of a Shared Interactive Surface into a Communal Space", In Proceedings of the ACM Conference on Computer Supported Cooperative Work, Nov. 2004, 10 Pages.
Buchenau, et al., "Experience Prototyping", In Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, Aug. 2000, pp. 424-433.
Butz, et al., "Applying the Peephole Metaphor in a Mixed-Reality Room", In Computer Graphics and Applications, vol. 26, No. 1, Jan. 2006, 8 Pages.
Butz, et al., "Searchlight-A Lightweight Search Function for Pervasive Environments", In Proceedings of the Second International Conference on Pervasive Computing, Apr. 2004, 6 Pages.
Cauchard, et al., "Steerable Projection: Exploring Alignment in Interactive Mobile Displays", In Personal and Ubiquitous Computing, vol. 16, No. 1, Jan. 1, 2012, 11 Pages.
Cook, et al., "Designers' Use of Paper and the Implications for Informal Tools", In Proceedings of the 1st Australia Conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future, Nov. 2005, 10 Pages.
Cotting, et al., "Adaptive Instant Displays: Continuously Calibrated Projections Using Per-Pixel Light Control", In Computer Graphics Forum. vol. 24. No. 3., Sep. 2005, pp. 705-714.
Cotting, et al., "Interactive Environment-Aware Display Bubbles", In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology, Oct. 2006, pp. 245-254.
Coutaz, et al., "Coupling Interaction Resources: An Analytical Model", In Proceedings of the Joint Conference on Smart Objects and Ambient Intelligence: Innovative Context-Aware services: UsaQes and TechnoloQies, Oct. 2005, pp. 183-188.
Dias, et al., "Tangible Interaction for Conceptual Architectural Design", In First IEEE International Workshop on Augmented Reality Toolkit, Sep. 2002, 9 Pages.
Dietz, et al., "DiamondTouch: A multi-user Touch Technology", In 14th Annual ACM Symposium on User Interface Software and Technology, Nov. 2001, 10 Pages.
Dodds, et al., "A Communication Task in HMO Virtual Environments: Speaker and Listener Movement", In Proceedings of the 23rd Annual Conference on Computer Animation and Social Agents, Jun. 2010, 4 Pages.
Ehnes, et al., "Projected Augmentation-Augmented Reality Using Rotatable Video Projectors", In Proceeding of Third IEEE and ACM International Symposium on Mixed and Augmented Reality, Nov. 2, 2004, 10 Pages.
El-Hakim, et al., "Sensor Based Creation of Indoor Virtual Environment Models", In Proceedings of the International Conference on Virtual Systems and MultiMedia, Sep. 10, 1997, 9 Pages.
El-Hakim, et al., "Two 3-D Sensors for Environment Modeling and Virtual Reality: Calibration and Multi-View Registration", In Proceedings of International Archives on Photogrammetry and Remote Sensing, vol. 31, Part B5, Commission V, Jul. 12, 1996, 10 Pages.
European Patent Office, International Search Report & Written Opinion for PCT Patent Application No. PCT/US2013/053878, Oct. 4, 2013, 11 Pages.
Everitt, Cass, "Projective Texture Mapping", Retrieved from <<https://web.archive.org/web/20031005055851/http://developer.nvidia.com/object/Projective-Texture-Mapping.html>>, Apr. 24, 2001, 7 Pages.
Fitzmaurice, et al., "Bricks: Laying the Foundations for Graspable User Interfaces", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, May 1, 1995, pp. 442-449.
Forelines, et al., "Direct-touch vs. Mouse Input for Tabletop Displays", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 2007, pp. 647-656.
Fuchs, et al., "Augmented Reality Visualization for Laparoscopic Surgery", Retrieved at << http://www.cs.odu.edu/.about.jrcrouch/papers/AR.sub.--Laparoscope.pdf >>, Lecture Notes in Computer Science, vol. 1496, Proceedings of the First International Conference on Medical Image Computing and Computer-Assisted Intervention, Oct. 11-13, 1998.
Gao, et al., "A Hemispherical Imaging Camera", Retrieved at <<vision.ai.uiuc.edu/publications/Gao-CVIU09.pdf>>, Computer Vision and Image Understanding Special Issue on Omni-directional Vision, Camera Networks, and Non-conventional Cameras, Feb. 2010, pp. 1-29.
Gao, et al., "A Hemispherical Imaging Camera", Retrieved at >, Computer Vision and Image Understanding Special Issue on Omni-directional Vision, Camera Networks, and Non-conventional Cameras, Feb. 2010, pp. 1-29.
Gao, et al., "Multi-User Interaction Using Handheld Projectors", In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, Oct. 7, 2007, 10 Pages.
Gargallo, et al., "Bayesian 3D Modeling from Images using Multiple Depth Maps", Retrieved at <<http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.143.8508&rep=rep- 1&type=pdf >>, CVPR, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), vol. 02, Jun. 20-26, 2005.
Gingichashvili, Sarah, "PlayAnywhere-Microsoft's Interactive Playground", Retrieved at << http://thefutureofthings.com/news/1037/playanywhere-microsofts-interactiv- e-playground.html >>, Nov. 2, 2007.
Grossman, et al., "An Interface for Creating and Manipulating Curves Using a High Degree-of-freedom Curve Input Device", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 5, 2003, pp. 185-192.
Guimbretiere, et al., "Fluid Interaction with High-Resolution Wall-Size Displays", In Proceedings of the 14th Annual ACM Symposium on User Interface Software and Technology, Nov. 2001, pp. 21-30.
Hailpern, et al., "TEAM STORM: Demonstrating an Interaction Model for Working with Multiple Ideas During Creative Group Work", In Proceedings of the 6th ACM SIGCHI Conference on Creativity & Cognition, Jun. 2007, pp. 193-202.
Haller, et al., "Shared Design Space: Sketching Ideas Using Digital Pens and a Large Augmented Tabletop Setup", In 16th International Conference on Artificial Reality and Telexistence, Nov. 2006, 12 Pages.
Harrison, et al., "OmniTouch: Wearable Multitouch Interaction Everywhere", In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Oct. 16, 2011, 10 Pages.
Henrysson, et al., "Face to face collaborative AR on mobile phones", In Proceedings of Fourth IEEE and ACM International Symposium Mixed and Augmented Reality, Oct. 5, 2005, pp. 80-89.
Hilliges, et al., "Interactions in the Air: Adding Further Depth to Interactive Tabletops", In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, Oct. 4, 2009, pp. 139-148.
Hinrichs, et al., "Examination of Text-Entry Methods for Tabletop Displays", In Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems, Oct. 2007, pp. 105-112.
Holman, et al., "Paper Windows: Interaction Techniques for Digital Paper", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 2005, pp. 591-599.
Hosokawa, et al., "Tangible Design Support System Using RFID Technology", In Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, Feb. 18, 2008, pp. 75-78.
Hu, et al., "Sensors and Data Fusion Algorithms in Mobile Robotics", In Technical Report CSM-422, University of Essex, Jan. 10, 2005, 13 Pages.
Jota, et al., "Constructing Virtual 3D Models With Physical Building Blocks", In Extended Abstracts on Human Factors in Computing Systems, May 2011, 6 Pages.
Kim, et al., "Video-Based Document Tracking: Unifying Your Physical and Electronic Desktops", In Proceedings of the Annual ACM Symposium on User Interface Software and Technology, Oct. 2004, pp. 99-107.
Kitamura, et al., "Real-time 3D Interaction with ActiveCube", In Conference on Human Factors in Computing Systems, Mar. 31, 2001, pp. 355-356.
Kjeldsen, et al., "Interacting with Steerable Projected Displays", In Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition, May 21, 2002, 6 Pages.
Klemmer, et al., "How Bodies Matter: Five Themes for Interaction Design", In Proceedings of the Designing Interactive Systems, Jun. 2006, pp. 140-149.
Klemmer, et al., "The Designers' Outpost: A Tangible ilnterface for Collaborative Web Site Design", In Proceedings of the 14th annual ACM Symposium on User Interface Software and Technology (UIST '01), 2001, 10 Pages.
Klemmer, et al., "The Designers' Outpost: A Task-Centered Tangible Interface for Web Site Information Design", Conference on Human Factors in Computing Systems (CHI), 2000, 2 pages.
Klemmer, Scott R., "Integrating Physical and Digital Interactions", In IEEE Computer Society, Oct. 2005, 4 Pages.
Koike, et al., "Integrating Paper and Digital Information on Enhanced Desk: A Method for Realtime Finger Tracking on an Augmented Desk System", In ACM Transaction on Computer-Human Interaction, vol. 8, Issue 4, Dec. 2001, pp. 307-322.
Koshimizu, et al., "SnapTable: Physical Handling for Digital Documents with Electronic Paper", In Proceedings of the Third Nordic Conference on Human-computer Interaction, Oct. 2004, pp. 401-404.
Labutov, et al., "Fusing Optical Flow and Stereo in a Spherical Depth Panorama Using a Single-Camera Folded Catadioptric Rig", Retrieved at <<http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5980103>>, IEEE International Conference on Robotics and Automation, May 9, 2011, pp. 3092-3097.
Labutov, et al., "Fusing Optical Flow and Stereo in a Spherical Depth Panorama Using a Single-Camera Folded Catadioptric Rig", Retrieved at >, IEEE International Conference on Robotics and Automation, May 9, 2011, pp. 3092-3097.
Lange, et al., "Insight lab: An Immersive Team Environment Linking Paper, Displays, and Data", In Proceedings of the Conference on Human Factors in Computing Systems, Jan. 1998, pp. 550-557.
Lavoie, et al., "Constructing 3D Virtual Reality Objects from 2D Images of Real Objects Using Nurbs", In Proceedings of IEEE Symposium on Human-Computer Interfaces and Measurement Systems, Virtual Environments, Jun. 25, 2007, pp. 117-124.
Leibe, et al., "The Perceptive Workbench: Toward Spontaneous and Natural Interaction in Semi-Immersive Virtual Environments", In Proceedings of the IEEE Virtual Reality Conference, Mar. 2000, 4 Pages.
Leibe, et al., "Toward Spontaneous Interaction with the Perceptive Workbench", In Proceedings of IEEE Computer Graphics and Applications, vol. 20, Issue: 6, Nov. 2000, pp. 54-65.
Leith Inger, et al., "Improving Menu Interaction for Cluttered Tabletop Setups with User-Drawn Path Menus", In Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer System, Oct. 2007, pp. 121-128.
Leiva, et al., "3D Reconstruction of a Static Indoor Environment by Fusion of Sonar and Video Data", In Proceedings of International Symposium on Intelligent Robotics Systems, Jul. 18, 2001, 10 Pages.
Levy, Steven, "Google Gets Transparent with Glass, Its Augmented Reality Project", Retrieved from <<http://www.wired.com/2012/04/epicenter-google-glass-ar/>>, Apr. 4, 2012, 4 Pages.
Liao, et al., "PapierCraft: A Command System for Interactive Paper", In Proceedings of the 18th Annual ACM symposium on User Interface Software and Technology, Oct. 23, 2005, 4 Pages.
Lin, et al., "DENIM: Finding a Tighter Fit Between Tools and Practice for Web Site Design", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 2000, pp. 510-517.
Luus, et al., "Optimization by Direct Search and Systematic Reduction of the Size of Search Region", In American Institute of Chemical Engineering Journal, vol. 19, Issue 4, Jul. 1973, 7 Pages.
Lzadi, et al., "KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera", In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Oct. 16, 2011, 10 Pages.
Manabe, et al., "Three Dimensional Measurement using Color Structured Patterns and Imaging Spectrograph", In Proceedings of 16th International Conference on Pattern Recognition, Aug. 11, 2002, pp. 649-652.
Matsushita, et al., "HoloWall: Designing a Finger; Hand, Body, and Object Sensitive Wall", In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology, Oct. 1997, pp. 209-210.
Meilland, et al., "An Augmented Spherical Visual Sensor for Large Scale 3D SLAM", Retrieved at <<http://www.open-phi.org/MS/publications/ICRA2011-submission.pdf>>, Information Signal Image Vision, Jan. 21, 2011, pp. 6.
Meilland, et al., "An Augmented Spherical Visual Sensor for Large Scale 3D SLAM", Retrieved at >, Information Signal Image Vision, Jan. 21, 2011, pp. 6.
Moggridge, Bill, "Designing Interactions", Table of Contents, Book published by MIT Press, Retrieved from <<http://www.designinginteractions.com/chapters>>, Retrieved on: Aug. 1, 2009, 1 Page.
Molyneaux, et al., "Cooperative Augmentation of Smart Objects with Projector-Camera Systems", In the 9th International Conference on Ubiquitous Computing, Sep. 16, 2007, 18 Pages.
Molyneaux, et al., "Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces", In Proceedings of the International Conference on Pervasive Computing, Jun. 18, 2012, 18 Pages.
Moraveji, et al., "Mischief: Supporting Remote Teaching in Developing Regions", In Proceedings of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, Apr. 2008, pp. 353-362.
Nakashima, et al., "An Integrated 2D-3D Work Environment for Cooperative Modeling", In Technical Report of the Institute of Image Information and Television Engineers, Feb. 2, 2004, pp. 41-46.
Newman, et al., "Sitemaps, Storyboards, and Specifications: A Sketch of Web Site Design Practice", In Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, Aug. 2000, pp. 263-274.
Ngan, et al., "Calibrating a Pan-Tilt Camera Head", In Image and Vision Computing Workshop, Dec. 1995, 6 Pages.
Nguyen, et al., "Depth image-based rendering from multiple cameras with 3D propagation algorithm", Retrieved at <<http://portal.acm.org/citation.cfm?id=1594116&dl=GUIDE&coll=GUIDE&CFID=85- 935589&CFTOKEN=23121094 >>, International Conference on Immersive Telecommunications, Proceedings of the 2nd International Conference on Immersive Telecommunications, May 27-29, 2009.
Office Action mailed Dec. 12, 2014 for U.S. Appl. No. 13/227,466, Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 26 pages.
Office Action mailed Dec. 14, 2015 for U.S. Appl. No. 13/227,466, Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 24 pages.
Office Action mailed Jan. 15, 2016 for U.S. Appl. No. 13/585,606, Wilson et al., "Wide Angle Depth Detection," filed Aug. 14, 2012, 12 pages.
Office Action mailed Jan. 29, 2014 for U.S. Appl. No. 13/227,466, Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 18 pages.
Office Action mailed Jul. 1, 2015 for U.S. Appl. No. 13/227,466 , Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 26 pages.
Office Action mailed Jul. 24, 2015 for U.S. Appl. No. 13/585,606 , Wilson et al., entitled "Wide Angle Depth Detection," filed Aug. 14, 2012, 10 pages.
Office Action mailed Jun. 20, 2013 for U.S. Appl. No. 13/227,466, Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 14 pages.
Office Action mailed Mar. 29, 2016 for U.S. Appl. No. 14/037,986, Wilson et al., entitled "Steerable Display System," filed Sep. 26, 2013, 27 pages.
Office Action mailed May 27, 2014 for U.S. Appl. No. 13/227,466, Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 20 pages.
Office Action mailed Nov. 18, 2015 for U.S. Appl. No. 14/037,986, Wilson et al., entitled "Steerable Display System," filed Sep. 26, 2013, 24 pages.
Office Action mailed Oct. 9, 2013 for U.S. Appl. No. 13/227,466, Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 17 pages.
Ohta, et al., "Share-Z: Client/Server Depth Sensing for See-Through Head Mounted Displays", Retrieved at << http://image-www.esys.tsukuba.ac.jp/research/publications/ohta.sub.--ISMR- .sub.--2001Mar.pdf >>, Presence: Teleoperators and Virtual Environments, vol. 11, No. 2, Apr. 2002.
Olsen, et al., "Spilling: Expanding Hand Held Interaction to Touch Table Displays", In Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer System, Oct. 2007, pp. 163-170.
Parker, et al., "TractorBeam: Seamless Integration of Local and Remote Pointing for Tabletop Displays", In Proceedings of Graphics Interface, May 2005, pp. 33-40.
Patten, et al., "Sensetable: A Wireless Object Tracking Platform for Tangible User Interfaces", In Proceedings of CHI, Mar. 2001, 8 Pages.
Pinhanez, et al., "Augmenting Reality with Projected Interactive Displays", In Proceedings of the International Symposium on Virtual and Augmented Architecture, Jun. 21, 2001, 9 Pages.
Pinhanez, et al., "The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces", In Proceedings of the 3rd International Conference on Ubiquitous Computing, Sep. 30, 2001, 18 Pages.
Piper, et al., "Illuminating Clay: A 3-D Tangible Interface for Landscape Analysis", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 2002, pp. 355-362.
Raffle, et al., "Topobo: A Constructive Assembly System with Kinetic Memory", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 25, 2004, pp. 647-654.
Raskar, et al., "iLamps: Geometrically Aware and Self-Configuring Projectors", In Special Interest Group on Computer Graphics and Interactive Techniques Conference, Jul. 2006, 10 Pages.
Raskar, et al., "Multi-Projector Displays Using Camera-Based Registration", In Proceedings of the Conference on Visualization, Oct. 24, 1999, 9 Pages.
Raskar, et al., "Shader lamps: Animating Real Objects with Image-Based Illumination", In Proceedings of the 12th Eurographics Workshop on Rendering Techniques, Jun. 26, 2001, 10 Pages.
Raskar, et al., "The Office of the Future: A Unified Approach to Image-Based Modeling and Spatially Immersive Displays", In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, Jul. 27, 1998, 10 Pages.
Rekimoto, et al., "Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, May 1999, pp. 378-385.
Rekimoto, et al., "Data Tiles: A Modular Platform for Mixed Physical and Graphical Interactions", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Mar. 2001, pp. 269-276.
Response filed Apr. 13, 2016, in response to Office Action mailed Jan. 15, 2016, for U.S. Appl. No. 13/585,606 , Wilson et al., entitled "Wide Angle Depth Detection," filed Aug. 14, 2012, 11 pages.
Response filed Apr. 29, 2014, in response to Office Action mailed Jan. 29, 2014, for U.S. Appl. No. 13/227,466 , Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 10 pages.
Response filed Aug. 27, 2014, in response to Office Action mailed May 27, 2014, for U.S. Appl. No. 13/227,466 , Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 13 pages.
Response filed Feb. 18, 2016, in response to Office Action mailed Nov. 18, 2015, for U.S. Appl. No. 14/037,986, Wilson et al., entitled "Steerable Display System," filed Sep. 26, 2013, 12 pages.
Response filed Feb. 29, 2016, in response to Office Action mailed Oct. 28, 2015, for U.S. Appl. No. 13/891,116, Benko et al., entitled "Immersive Display With Peripheral Illusions," filed May 9, 2013, 13 pages.
Response filed Jan. 9, 2014, in response to Office Action mailed Oct. 9, 2013, for U.S. Appl. No. 13/227,466 , Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 9 pages.
Response filed Jul. 14, 2015, in response to Office Action mailed Jan. 14, 2015, for U.S. Appl. No. 13/891,116, Benko et al., entitled "Immersive Display With Peripheral Illusions," filed May 9, 2013, 10 pages.
Response filed Mar. 12, 2015, in response to Office Action mailed Dec. 12, 2014, for U.S. Appl. No. 13/227,466 , Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 13 pages.
Response filed Mar. 14 2016, in response to Office Action mailed Dec. 14, 2015, for U.S. Appl. No. 13/227,466 , Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 14 pages.
Response filed Oct. 1, 2015, in response to Office Action mailed Jul. 1, 2015, for U.S. Appl. No. 13/227,466 , Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 14 pages.
Response filed Oct. 26, 2015, in response to Office Action mailed Jul. 24, 2015, for U.S. Appl. No. 13/585,606 , Wilson et al., entitled "Wide Angle Depth Detection," filed Aug. 14, 2012, 14 pages.
Response filed Sep. 20, 2013, in response to Office Action mailed Jun. 20, 2013, for U.S. Appl. No. 13/227,466 , Wilson, Andrew, entitled "Depth Camera As a Touch Sensor," filed Sep. 7, 2011, 8 pages.
Response filed Sep. 30, 2015, in response to Office Action mailed Apr. 27, 2015, for U.S. Appl. No. 14/037,986, Wilson et al., entitled "Steerable Display System," filed Sep. 26, 2013, 12 pages.
Response to "Examination Report" Received in European Patent Application No. 11747870.1, Submitted Date: Jun. 25, 2014, 10 Pages.
Scheibe, et al., "Data Fusion and Visualization of Panoramic Images and Laser Scans", In ISPRS Working Group V/1, Panoramic Photogrammetry Workshop, Feb. 19, 2004, 8 Pages.
Segal, et al., "Fast Shadows and Lighting Effects Using Texture Mapping", In Proceedings of the 19th Annual Conference on Computer Graphics and Interactive Techniques, Jul. 1, 1992, pp. 249-252.
Sequeira, et al., "3D Reconstruction of Indoor Environments", In Proceedings of International Conference on Image Processing, vol. 1, Sep. 16, 1996, 4 Pages.
Shen, et al., "DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Apr. 2004, pp. 167-174.
Sodhi, et al., "Kinect-Projector Calibration", CS 498 Computational Photography-Final Project, University of Illinois at Urbana-Champaign, Dec. 14, 2010, 11 Pages.
Sodhi, et al., "Lightguide: Projected Visualizations for Hand Movement Guidance", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, May 5, 2012, 10 Pages.
Sokoler, et al., "Physically Embodied Video Snippets Supporting Collaborative Exploration of Video Material During Design Sessions", In Proceedings of the Second Nordic Conference on Human-Computer Interaction, Oct. 2002, pp. 139-148.
Song, et al., "Use of LCD Panel for Calibrating Structured-Light-Based Range Sensing System", In Proceedings of IEEE Transactions on Instrumentation and Measurement, vol. 57, Issue: 11, Jun. 6, 2008, pp. 2623-2630.
Starner, et al., "The Perceptive Workbench: Computer-Vision-Based Gesture Tracking, Object Tracking, and 3D Reconstruction for Augmented Desks", In Machine Vision and Applications, Apr. 2003, 13 Pages.
Subrahmanian, et al., "Boundary Objects and Prototypes at the Interfaces of Engineering Design", In Journal of Computer Supported Cooperative Work, vol. 12 Issue 2, ACM Abstract Provided Only, May 2003, 2 Pages.
Taguchi, et al., "Axial-Cones: Modeling Spherical Catadioptric Cameras for Wide-Angle Light Field Rendering", Retrieved at <<http://www.umiacs.umd.edu/˜aagrawal/sig10Asia/AxialConeSIGAsia10LowRes.pdf>>, The 3rd ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, Dec. 3, 2010, pp. 8.
Taguchi, et al., "Axial-Cones: Modeling Spherical Catadioptric Cameras for Wide-Angle Light Field Rendering", Retrieved at >, The 3rd ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia, Dec. 3, 2010, pp. 8.
Tang, et al., "A Framework for Understanding the Workspace Activity of Design Teams", In Proceedings of the ACM Conference on Computer-Supported Cooperative Work, Jan. 1988, pp. 244-249.
Tang, et al., "Augmented Reality Systems for Medical Applications", In Proceedings of Engineering in Medicine and Biology Magazine, May 1998, pp. 49-58.
Tsai, et al., "A New Technique for Fully Autonomous and Efficient 3D Robotics Hand/Eye Calibration", In IEEE Transactions on Robotics and Automation, vol. 5, No. 3, Jun. 1989, 14 Pages.
U.S. Appl. No. 14/742,458, Benko et al., entitled "Complementary Augmented Reality," filed Jun. 17, 2015, 66 pages.
Ullmer, et al., "The metaDESK: Models and Prototypes for Tangible User Interfaces", In Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology, Oct. 1997, pp. 223-232.
Underkoffler, et al., "Emancipated Pixels: Real-World Graphics in the Luminous Room", In Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, ACM Press/Addison-Wesley Publishing Co., Jul. 1, 1999, 8 Pages.
Underkoffler, et al., "Urp: A Luminous-Tangible Workbench for Urban Planning and Design", In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, May 1999, pp. 386-393.
Wellner, Pierre, "Interacting with Paper on the Digital Desk", In Communications of the ACM-Special Issue on Computer Augmented Environments: Back to the Real World, vol. 36, Issue 7, Jul. 1993, pp. 87-96.
Wesche, et al., "Applications of the Responsive Workbench", In Proceedings of IEEE Computer Graphics and Applications, vol. 17, Issue: 4, Jul. 1997, pp. 10-15.
Willis, et al., "SideBySide: Ad-hoc Multi-user Interaction with Handheld Projectors", In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, Oct. 16, 2011, 10 Pages.
Wilson, Andrew D., "Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction", Retrieved at << http://research.microsoft.com/en-us/um/people/awilson/publications/Wilson- Tabletop2007/WilsonTabletop2007.html >>, In Proceedings of the 2nd IEEE International Workshop on Horizontal Interactive Human-Computer Systems (Tabletop), 2007.
Wilson, Andrew D., "Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction," In Proceedings of Second Annual IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP 2007), Oct. 10-12, 2007, pp. 201-204.
Wilson, D. A., "Simulating Grasping Behavior on an Imaging Interactive Surface", In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces, Oct. 23, 2009, pp. 125-132.
Wilson, et al., "Bringing Physics to the Surface", In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, Oct. 2008, pp. 67-76.
Wilson, et al., "Combining Multiple Depth Cameras and Projectors for Interactions on, above and between Surfaces", In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology, Oct. 3, 2010, pp. 273-282.
Wilson, et al., "FiowMouse: A Computer Vision-Based Pointing and Gesture Input Device", In Human-Computer Interaction-INTERACT, Sep. 2005, 14 Pages.
Wilson, et al., "Steerable Augmented Reality with the Beamatron", In Proceedings of the 25th annual ACM symposium on User Interface Software and Technology (USIT), Oct. 7, 2012, 10 Pages.
Wu, et al., "Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces", In Proceedings of the First IEEE International Workshop on Horizontal Interactive Human-Computer Systems, Jan. 2006, 8 Pages.
Zhang, et al., "A Flexible New Technique for Camera Calibration", In IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, No. 11, Nov. 1, 2000, 5 Pages.
Zhang, et al., "Visual Panel: Virtual Mouse, Keyboard and 3D Controller with an Ordinary Piece of Paper", In Proceedings of the Workshop on Perceptive User Interfaces, Nov. 2001, 8 Pages.
Zhou, et al., "Multi-Projector Display with Continuous Self-Calibration", Retrieved at << http://vis.uky.edul.about.wangl/Research/Publication/2008/continuous.sub.- --display.sub.--procams08.pdf >>, International Conference on Computer Graphics and Interactive Techniques, Proceedings of the 5th ACM/IEEE International Workshop on Projector camera systems, Aug. 10, 2008.
US8730309B2 (en) 2014-05-20 grant
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WILSON, ANDREW DAVID;BENKO, HRVOJE;SIGNING DATES FROM 20140711 TO 20140722;REEL/FRAME:033364/0936