EXHIBIT 4

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 1 of 22
c12) United States Patent 

Eraker et al. 

(54) IMAGE-BASED RENDERING OF REAL 

SPACES 

(71) Applicant: Appliance Computing III, Inc., 

Seattle, WA (US) 

(72) Inventors: David Eraker, Seattle, WA (US); 

Aravind Kalaiah, San Jose, CA (US); 

Robert McGarty, Seattle, WA (US) 

(73) Assignee: Appliance Computing III, Inc., 

Seattle, WA (US) 

( *) Notice: Subject to any disclaimer, the term ofthis 

patent is extended or adjusted under 35 

U.S.C. 154(b) by O days. 

This patent is subject to a terminal disclaimer. 

(21) Appl. No.: 16/681,287 

(22) Filed: Nov. 12, 2019 

Related U.S. Application Data 

(63) Continuation of application No. 16/130,555, filed on 

Sep. 13, 2018, now Pat. No. 10,510,111, which is a 

(51) Int. Cl. 

G06Q 30/06 

G09G 5114 

(52) U.S. Cl. 

(Continued) 

(2012.01) 

(2006.01) 

(Continued) 

CPC ..... G06Q 30/0643 (2013.01); G06F 3/04815 

(2013.01); G06F 3/04842 (2013.01); 

(Continued) 

( 58) Field of Classification Search 

None 

See application file for complete search history. 

102 

I 1111111111111111 1111111111 111111111111111 1111111111 11111 1111111111 11111111 

USO 10592973B 1 

(10) Patent No.: US 10,592,973 Bl 

(45) Date of Patent: *Mar. 17, 2020 

(56) 

CA 

CN 

References Cited 

U.S. PATENT DOCUMENTS 

6,760,026 B2 

6,903,738 B2 

7/2004 Li et al. 

6/2005 Pfister et al. 

(Continued) 

FOREIGN PATENT DOCUMENTS 

2587644 1/2015 

1242851 1/2000 

(Continued) 

OTHER PUBLICATIONS 

U.S. Appl. No. 61/502,427, filed Jun. 29, 2011, Bell et al. 

(Continued) 

Primary Examiner - Frank S Chen 

(74) Attorney, Agent, or Firm - Ellen M. Bierman; Lowe 

Graham Jones PLLC 

(57) ABSTRACT 

Under an embodiment of the invention, an image capturing 

and processing system creates 3D image-based rendering 

(IBR) for real estate. The system provides image-based 

rendering of real property, the computer system including a 

user interface for visually presenting an image-based rendering of a real property to a user; and a processor to obtain 

two or more photorealistic viewpoints from ground truth 

image data capture locations; combine and process two or 

more instances of ground truth image data to create a 

plurality of synthesized viewpoints; and visually present a 

viewpoint in a virtual model of the real property on the user 

interface, the virtual model including photorealistic viewpoints and synthesized viewpoints. 

23 Claims, 8 Drawing Sheets 

100 

~ 

108 

Real Estate Web 

Page with Imagebased Rendering 

112 

Database 

110 

104 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 2 of 22
US 10,592,973 Bl 

Page 2 

Related U.S. Application Data 

continuation of application No. 15/832,653, filed on 

Dec. 5, 2017, now Pat. No. 10,102,673, which is a 

continuation of application No. 14/525,057, filed on 

Oct. 27, 2014, now Pat. No. 9,836,885. 

(60) Provisional application No. 61/895,978, filed on Oct. 

25, 2013. 

(51) 

(52) 

Int. Cl. 

G06F 3/0484 

G06F 3/0481 

G06F 161951 

G06T 17100 

G06T 11160 

G06T 19100 

G06Q 50/16 

U.S. Cl. 

(2013.01) 

(2013.01) 

(2019.01) 

(2006.01) 

(2006.01) 

(2011.01) 

(2012.01) 

CPC ....... G06F 161951 (2019.01); G06Q 30/0623 

(2013.01); G06Q 50/16 (2013.01); G06T 11160 

(2013.01); G06T 17100 (2013.01); G06T 

191003 (2013.01); G09G 5114 (2013.01); G06T 

2210/04 (2013.01); G06T 2219/028 (2013.01); 

G09G 2340/12 (2013.01) 

(56) References Cited 

U.S. PATENT DOCUMENTS 

6,968,973 B2 11/2005 Uyttendaele et al. 

7,126,603 B2 10/2006 Aliaga et al. 

7,463,280 B2 12/2008 Steuart, III 

7,570,261 Bl 8/2009 Edecker et al. 

7,834,883 B2 11/2010 Adams 

8,140,421 Bl 3/2012 Humphries et al. 

8,207,964 Bl 6/2012 Meadow et al. 

8,213,916 Bl 7/2012 Yankovich et al. 

8,339,394 Bl 12/2012 Lininger 

8,346,017 B2 1/2013 Aguera et al. 

8,635,104 B2 1/2014 Adams 

8,681,151 B2 3/2014 Coombe et al. 

8,861,840 B2 10/2014 Bell et al. 

8,861,841 B2 10/2014 Bell et al. 

8,879,828 B2 11/2014 Bell et al. 

8,890,863 Bl 11/2014 Lininger 

9,013,505 Bl 4/2015 Thornton 

9,087,401 Bl 7/2015 Zheng et al. 

9,098,908 B2 8/2015 Kirk et al. 

9,165,410 Bl 10/2015 Bell et al. 

9,171,405 Bl 10/2015 Bell et al. 

9,324,190 B2 4/2016 Bell et al. 

9,361,943 B2 6/2016 Thrun 

9,396,586 B2 7/2016 Bell et al. 

9,489,775 Bl 11/2016 Bell et al. 

9,786,097 B2 10/2017 Bell et al. 

9,888,215 B2 2/2018 Sankar et al. 

9,953,111 B2 4/2018 Bell et al. 

10,055,876 B2 8/2018 Ford et al. 

10,127,722 B2 11/2018 Shakib et al. 

10,139,985 B2 11/2018 Mildrew et al. 

2002/0065739 Al 5/2002 Florance et al. 

2002/0087389 Al 7/2002 Sklarz et al. 

2002/0113791 Al 8/2002 Li et al. 

2003/0083957 Al 5/2003 Olefson 

2004/0012602 Al 1/2004 Mech 

2004/0019517 Al 1/2004 Sennott 

2004/0240741 Al 12/2004 Aliaga et al. 

2006/0085234 Al 4/2006 Cagan 

2006/0105342 Al 5/2006 Villena et al. 

2006/0218003 Al 9/2006 Snyder 

2006/0218005 Al 9/2006 Villena et al. 

2007/0127823 Al 6/2007 Seeber 

2007/0198278 Al 8/2007 Cheng et al. 

2008/0106593 Al 5/2008 Arfvidsson et al. 

2009/0031246 Al 1/2009 Cowtan et al. 

2009/0189983 Al 7/2009 Brumfield et al. 

2010/0214284 Al 8/2010 Rieffel et al. 

2010/0251101 Al 9/2010 Haussecker et al. 

2011/0276499 Al 11/2011 Walsh et al. 

2011/0279445 Al 11/2011 Murphy et al. 

2011/0279453 Al 11/2011 Murphy et al. 

2012/0179983 Al 7/2012 Lemire 

2012/0278465 Al 11/2012 Johnson 

2013/0073388 Al 3/2013 Heath 

2013/0095920 Al 4/2013 Pati ej unas et al. 

2013/0174059 Al 7/2013 Van Wie et al. 

2013/0222369 Al 8/2013 Huston et al. 

2013/0321396 Al 12/2013 Kirk et al. 

2013/0321575 Al 12/2013 Kirk et al. 

2013/0321586 Al 12/2013 Kirk et al. 

2013/0321589 Al 12/2013 Kirk et al. 

2013/0321593 Al 12/2013 Kirk et al. 

2014/0053077 Al 2/2014 Unnikrishnan et al. 

2014/0092281 Al 4/2014 Nisenzon et al. 

2014/0125658 Al 5/2014 Bell et al. 

2014/0125767 Al 5/2014 Bell et al. 

2014/0125768 Al 5/2014 Bell et al. 

2014/0125769 Al 5/2014 Bell et al. 

2014/0125770 Al 5/2014 Bell et al. 

2014/0198127 Al 7/2014 Ying 

2014/0285486 Al 9/2014 Chang et al. 

2014/0301645 Al 10/2014 Mattila 

2015/0186953 Al 7/2015 Gross 

2015/0268043 Al 9/2015 McFadden et al. 

2017/0069142 Al 3/2017 Elmekies 

FOREIGN PATENT DOCUMENTS 

JP 11-259672 9/1999 

JP 2004-265396 9/2004 

OTHER PUBLICATIONS 

U.S. Appl. No. 61/603,221, filed Feb. 24, 2012, Bell et al. 

U.S. Appl. No. 61/663,265, filed Jun. 22, 2012, Bell. 

"Competition and Real Estate," The United States Department of 

Justice, updated Jan. 12, 2016, retrieved from the Internet at 

http://www.justice.gov/atr/public/real_estate/, on Aug. 22, 2019, 2 

pages. 

"Enforcing Antitrust Laws in the Real Estate Industry," The United 

States Department of Justice, updated Jul. 2, 2015, retrieved from 

the Internet at http://www.justice.gov/atr/public/real_estate/enforce. 

html, on Aug. 22, 2019, 2 pages. 

"Google Street View gets peek inside Sheffield university buildings," article amended on Apr. 5, 2013, retrieved from the Internet 

through theguardian.com on Aug. 2, 2019, 3 pages. 

"How Redfin Helped Optimize Matterport Technology for Real 

Estate," Matterport, retrieved from the Internet through matterport. 

com, on Aug. 22, 2019, 2 pages. 

"Improving Competition in Real Estate Transactions," Policy Roundtables, Competition Law & Policy, OECD, 2007, retrieved from the 

Internet at http://www.oecd.org/regreforrn/ sectors/397 48266 .pdf, on 

Aug. 22, 2019, 245 pages. 

"Laser Scanner for Fast and Exact Indoor and Outdoor Measurements in Three Dimensions: Simply at Your Fingertips," Focus 

Laser Scanner Series, FARO, retrieved from the Internet at http:// 

www.faro.com/ en-us/products/3 d-surveying/faro-focus3 d/ 

overview, on Aug. 22, 2019, 4 pages. 

"Machine Vision-Area Scan Cameras," FLIR Systems, Inc., retrieved 

from the Internet at http://ww2.ptgrey.com/products/ladybug5/, on 

Aug. 22, 2019, 2 pages. 

"Matterport Raises $16 Million in Series B Financing to Lead 3D 

Expansion," Matterport Press Release, Jul. 2014, Mountain View, 

CA, retrieved from the Internet through matterport.com, on Aug. 2, 

2019, 4 pages. 

"Quick Real Estate Statistics," National Association of Realtors, 

May 11, 2018, retrieved from the Internet at http://www.realtor.org/ 

field-guides/field-guide-to-quick-real-estate-statistics, on Aug. 22, 

2019, 5 pages. 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 3 of 22
US 10,592,973 Bl 

Page 3 

(56) References Cited 

OTHER PUBLICATIONS 

"The great realtor rip-off," The Economist, May 5, 2012, retrieved 

from the Internet at http://www.economist.com/node/21554204, on 

Aug. 22, 2019, 2 pages. 

Baker, Natasha, "New apps help shoppers identify items using 

image recognition," Reuters, 2013, retrieved from the Internet at 

https://www.reuters.com/article/us-apps-pounce/new-apps-helpshoppers-identify-items-using-image-recognitionidUSBRE97P06J20130826, on Aug. 22, 2019, 5 pages. 

Bernheim, B. Douglas et al., "Do Real Estate Brokers Add Value 

When Listing Services Are Unbundled," NBER Working Paper 

Series, Working Paper 13796, National Bureau of Economic Research, 

Cambridge, MA, Feb. 2008, retrieved from the Internet at http:// 

www.nber.org/papers/w13796.pdf?new_window=l, onAug. 22, 2019, 

32 pages. 

Debevec, Paul et al. "Efficient View-Dependent Image-Based Rendering With Projective Texture-Mapping," G. Drettakis et al. (eds.), 

Rendering Techniques' 98, Springer-Verlag Wien, Vienna, 1998, 13 

pages. 

Elfelt, Joseph, "#1----Google maps now shows property parcel lines 

for many areas," Blog entry, www.MappingSupport.com, Redmond, 

Washington, Oct. 19, 2009, available at https://activerain.com/ 

droplet/yfz, 10 pages. 

Garmin Oregon 450t Handheld GPS Navigator, by Garmin, retrieved 

from the Internet at http://www.amazon.com/Garmin-Oregon-450tHandheld-N avigator/dp/B00313JX56/ref~sr_l_l27Fgps&le~UTF8 

&aid~l380814616&sFl-12, on Aug. 22, 2019, 6 pages. 

He, Li-Wei et al., "Rendering with Concentric Mosaics," Microsoft, 

Association for Computing Machinery, Inc., Jan. 1999, 8 pages. 

Levitt et al., "Antitrust Implications of Outcomes When Home 

Sellers Use Flat-Fee Real Estate Agents," Project Muse, BrookingsWharton Papers on Urban Affairs, 2008, pp. 47-93, retrieved from 

the Internet at http://muse.jhu.edu/journals/brookings- wharton papers 

on urban affairs/v2008/2008.levitt.pdf, on Aug. 22, 2019. 

Peck, Emily, "In Real Estate, a Picture Is Worth $1,000 or More," 

The Wall Street Journal, Oct. 4, 2010, retrieved from the Internet at 

http://blogs.wsj.com/developments/2010/ 10/04/in-real-estate-apicture-is-worth-1000-or-more/, on Aug. 22, 2019, 2 pages. 

Sankar et al. "Capturing Indoor Scenes with Smartphones," UIST' 

12, Oct. 7-10, 2012, Cambridge, Massachusetts, 9 pages. 

Schwab, Klaus (ed.), "The Global Competitiveness Report 2011-

2012," World Economic Forum, Geneva, Switzerland, 2011, retrieved 

from the Internet at http://www3.weforum.org/docs/WEF _GCR_ 

Report_2011-12.pdf, on Aug. 22, 2019, 544 pages. 

Shum, Heung-Yeung et al., "A Review of Image-based Rendering 

Techniques," Microsoft, In Visual Communications and Image 

Processing 2000, Ngan, King N. et al. (eds.), Proceedings of SPIE, 

vol. 4067, 2000, 12 pages. 

"Stanley 77-018 IntelliMeasure® Distance Estimator" fact sheet, 

Item No. WBB983318, globalindustrial.com, retrieved from the 

Internet at http://www.globalindustrial.com/p/tools/test-measurement/ 

Gauge-Block-Ball-Sets/77-018-intellimeasure-distance-estimator? 

infoParam.campaign 1 d~T9 A&gclid ~CNyrut3 _ -

rkCFRDZOgodgRMAxw, on Aug. 22, 2019, 2 pages. 

Szeliski, Richard, "Computer Vision: Algorithms and Applications," Draft, Springer, Sep. 3, 2010, Chapter 13, "Image-based 

rendering," pp. 619-654. 

Uyttendaele, Matthew et al., "High-quality Image-based Interactive 

Exploration of Real-World Environments," Technical Report, MSRTR-2003-61, Microsoft Corporation, Redmond, Washington, Oct. 

2003, 34 pages. 

Videos of Photosynth (printout of page where located), retrieved on 

the Internet at http:/ /photosynth.net/view.aspx?cid~24 ld5bf8-d92f49f8-b94b-8d5fea6ela2c on Aug. 22, 2019, 2 pages. 

Zhang, Cha et al., "A Survey on Image-Based RenderingRepresentation, Sampling and Compression," Technical Report 

AMP 03-03, Electrical & Computer Engineering, Carnegie Mellon 

University, Pittsburgh, Pennsylvania, Jun. 2003, 45 pages. 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 4 of 22
U.S. Patent 

n 

QJ 

L 

::J 

b.O 

LL 

Mar.17,2020 

N 

0 

n 

Sheet 1 of 8 

Q) 

V) 

ro 

..0 

ro 

+-' 

ro 

0 

..o d, ~o 

OJ tl.O ·- n $ ro !....n E w 

Q) - "'O 

+-' ..C C 

ro +-' OJ 

t, ·- 0:: 

LU $ "'O 

ro OJ OJ 

Q) tl.O V) 

,..,., ro ro u... CL ..0 

US 10,592,973 Bl 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 5 of 22
Figure 2 

200 

PROCESSING 

UNIT 204 

SYSTEM M 

206 

MEMORY 

~ I RAM ~=r 212 

ROM 210 

I V) 

::::) 

CO I I 

VIDEO ADAPTER 218 

SERIAL PORT 

2081 l 1NTERFAc~;u1d 

NETWORK 

ADAPTE'2_56! 

WIRELESS 

TRANSCEIVER 

202 

;--------------------------------------------------------· I I 

! OPERATING SYSTEM 230 i I I I I ll..--------------------------------------------------------.1 

I APPLICATIONS 232 ! I I I I ._ ________________________________________________________ .. 

I I 

i MODULES 234 i I I I I L ........ - ....................................................................................................... .1 

___ .,..., .. _.,. ..... •••••• .. • .. •••••• .. ••••• .. •••• .. ••••• I 

I I 

! DATA 236 ! I I I I L--------------------------------------------------------.1 

DISPLAY 

220 

WLAN 

KEYBOARD 

244 

REMOTE COMPUTERS 

248 

250 

MEMORY STORAGE 

e 

• 

00 

• 

~ 

~ 

~ 

~ 

= ~ 

~ ~ 

:-: 

.... 

'"-....J 

N 

0 

N 

0 

rJJ 

('D =-

('D 

..... 

N 

0 

.... 

QO 

d 

r.,;_ 

"'""' '"= UI 

\0 

N 

\0 

-....l 

w 

= "'""' 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 6 of 22
U.S. Patent 

(Y) 

cu 

L 

::J 

tlD 

LL 

Mar.17,2020 

0 

0 

("('} 

ro 

+-' 

ro 

0 

Q) 

!.... 

::J 

+-' 

a. 

ro 

u 

n 

0 

("('} 

~ 

Sheet 3 of 8 US 10,592,973 Bl 

N ("('} 

0 0 

("('} ("('} 

ro "'O 

+-' Q) tl.O ro V) C 0 ro 

..0 !.... 

V) Q) V) ~ d, "'O Q) 

u tl.O C 

0 ro Q) 

!.... E 0:: 

CL -

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 7 of 22
U.S. Patent Mar.17, 2020 Sheet 4 of 8 

~ 

cu of L 

::J 0 

tlD o::::t" 

·-LL 

o::::t" 

0 

o::::t" 

US 10,592,973 Bl 

00 

0 

o::::t" 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 8 of 22
Define sp~tial N Sample RGB image r---,-.. 

boundaries 502 data Create panoramas 

I ___......; Determine camera 

~ I geometry 508 

Generate dense 

representation 

of a 3D model 

geometry 

510 

504 

506 

Determine Generate Generate a 

i--.1 appropriate ..,__.. parcel map and ,..___. geometric proxy 

geometric proxy floor plan from for the 3D r---,-.. 

3D data model 

512 514 

Figure 5 

516 

Render3D 

scene 

518 

e 

• 

00 

• 

~ 

~ 

~ 

~ 

= ~ 

~ ~ 

:-: 

.... 

'"-....J 

N 

0 

N 

0 

rJJ 

('D =-

('D 

..... 

Ul 

0 

.... 

QO 

d 

r.,;_ 

"'""' '"= UI 

\0 

N 

\0 

-....l 

w 

= "'""' 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 9 of 22
+ 1 

Input images Input IR and/or 

from various Laser data I I 

Meta & detailed 

devices 1501 i 

610 Labels I 615 I 

• • T 

Image processing: 

Register RGB to Point, 10, 2D & 3D 

Depth data I 611 1 

Geometric Proxies 

r+ HDR, noise reduction 

etc. ~ ' t J l 

$ 

Q) 

• ICP-Based I 612 I 

·;; 

I Q) 

Stitched! 603 

Registration of f-+ .... 3D Point 

I 

0.. -

Depth data 

.... 

Panoramas ~ 

Clou~ 613 

~ 

-.,. ~ 

t 

Feature Pano Neighborhood 

Pano BundleE 

Matching Graph (auto and/or __,.. 

Spanning 

tree~ 

Adjustment 

between f-+ human assiste1l·6·os····1 

Panos I 604 I I ~ 

Pairwise Pano Registration ... I 606 I 

Figure 6 

~ 

--+ Image Based Rendering 

614 l 616 ] 

.H. J l 

~ 

Pa no [~609] ... extrinsics 

e 

• 

00 

• 

~ 

~ 

~ 

~ 

= ~ 

~ ~ 

:-: 

.... 

'"-....J 

N 

0 

N 

0 

rJ'1 

('D =-

('D 

..... 

O'I 

0 

.... 

QO 

d 

r.,;_ 

"'""' '"= UI 

\0 

N 

\0 

-....l 

w 

= "'""' 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 10 of 22
U.S. Patent 

~ 

cu 

L 

::J 

tlD 

LL 

Mar.17,2020 

o::::t" 

0 

r--,. 

I -

t_ __ ...... 

Sheet 7 of 8 

N 

0 

r--,. 

US 10,592,973 Bl 

I.O 

0 

r--,. 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 11 of 22
User requests view 

point 801 

Render Panorama 

or other sampled 

viewpoint 803 

Figure 8 

View-dependent 

texture map 805 

'" 

No view-point 

synthesis 

Pure synthetic 

804 

rendering from 

geometric proxies 

806 

'" 

,1r 

Blended/Selected 

rendering 808 

Light-field 

rendering 

807 

, " 

e 

• 

00 

• 

~ 

~ 

~ 

~ 

= ~ 

~ ~ 

:-: 

.... 

'"-....J 

N 

0 

N 

0 

rJJ 

('D =-

('D 

..... 

QO 

0 

.... 

QO 

d 

r.,;_ 

"'""' '"= UI 

\0 

N 

\0 

-....l 

w 

= "'""' 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 12 of 22
US 10,592,973 B 1 

1 

IMAGE-BASED RENDERING OF REAL 

SPACES 

CROSS-REFERENCE TO RELATED 

APPLICATIONS 

2 

photos or videos. However, the location of the viewpoint is 

still restricted to the location at which the panorama photo 

was taken. 

From the buyer's perspective, real estate websites suffer 

5 from numerous problems at present. First among these is the 

typical gallery of photographs of the property. Usually, the 

photographs are taken by the real estate agent or homeowner 

and are intended to highlight the positive attributes of the 

property while minimizing any negative attributes. The 

This application is a continuation of U.S. patent application Ser. No. 16/130,555 filed Sep. 13, 2018; which is a 

continuation of U.S. patent application Ser. No. 15/832,653 

filed Dec. 5, 2017 and issued as U.S. Pat. No. 10,102,673; 

which is a continuation of U.S. patent application Ser. No. 

14/525,057 filed Oct. 27, 2014 and issued as U.S. Pat. No. 

9,836,885, which claims the benefit of U.S. Provisional 

Patent Application No. 61/895,978 filed Oct. 25, 2013. This 15 

application is also related to U.S. patent application Ser. No. 

14/525,052, filed Oct. 27, 2014, entitled "USER INTERFACE FOR IMAGE-BASED RENDERING OF VIRTUAL 

TOURS", U.S. patent application Ser. No. 14/525,059, filed 

Oct. 27, 2014, entitled "IMAGE-BASED RENDERING OF 

VIRTUAL MODELS OF REAL SPACES", and U.S. patent 

application Ser. No. 14/525,060, filed Oct. 27, 2014, entitled 

"IMAGE-BASED RENDERING OF THREE DIMENSION 

10 photo galleries are usually navigated linearly, by proceeding 

from one two-dimensional photograph to the next. Lastly, 

the viewer is restricted to the viewpoint of the available 

photographs. To get a different viewpoint, the buyer must 

visit the property in person. 

Potential home buyers suffer from a lack of real estate 

information and tools. Many real estate brokerage websites 

provide some photographs of the properties in their listings. 

However, these websites remain surprisingly poor at providing comprehensive visual and spatial information about 

20 properties. 

There is a need for a system that overcomes limitations of 

the current methods of creating virtual models of real 

properties, as well as providing additional benefits. 

GEOMETRIES". All of the above U.S. patent applications 

are fully incorporated herein by reference. 25 SUMMARY 

TECHNICAL FIELD 

This disclosure relates generally to methods and systems 

for image-based rendering and, more particularly, relates to 

the creation and rendering of three dimensional geometry 

combined with images of real world scenes. 

BACKGROUND 

Real estate websites typically have a description of the 

real estate listed for sale on the website. The description 

often includes pictures of the real estate in addition to a 

written description of the property. Photographs of the 

property are helpful and an improvement over a text description. However, without physically visiting the property to 

look at it, the property can only be seen from the viewpoint 

of the photographer and lack spatial navigation. So photographs, while helpful, have not eliminated the need to 

physically go to the property and understand how it is laid 

out. 

Some real estate websites have posted video tours of their 

listed properties, which can be even more helpful than 

photographs. Often these videos are created by a realtor or 

home-owner who walks through the home while recording 

with a video camera. The quality of those videos is generally 

not good, so some realtors will use professional videographers to create a more polished and professional video. 

Unfortunately, videos are likewise limited in the sense that 

the viewer can only see the property from the viewpoint of 

the videographer. Also, the viewer does not have the ability 

to tour the property on his own route because the videos 

have a predetermined path through the house. 

In an attempt to allow the viewer to have more control 

over what he can observe, some real estate websites have 

used panorama cameras to take panoramic photos of the 

properties. This has the advantage that, from the location at 

which the panorama was taken, the viewer can "rotate" his 

viewpoint, thus observing the entire panoramic photo. These 

panoramas have the advantage that they give a more "inperson" feel to looking at the property than traditional 

A brief summary of some embodiments and aspects of the 

invention are first presented. Some simplifications and omissions may be made in the following summary; the summary 

30 is intended to highlight and introduce some aspects of the 

disclosed embodiments, but not to limit the scope of the 

invention. Thereafter, a detailed description of illustrated 

embodiments is presented, which will permit one skilled in 

the relevant art to make and use aspects of the invention. 

35 One skilled in the relevant art can obtain a full appreciation 

of aspects of the invention from the subsequent detailed 

description, read together with the Figures, and from the 

claims (which follow the detailed description). 

In one embodiment of the invention, a three dimensional 

40 model of a real scene is constructed from image data such as 

spherical panoramic photos, according to a plurality of 

image-based rendering (IBR) algorithms. Rather than use 

one image-based rendering algorithm throughout the three 

dimensional model, the location of the viewpoint in the three 

45 dimensional model may be a factor in choosing among 

several image based rendering algorithms. Viewpoint locations having real image data will result in photorealistic or 

near photorealistic panoramas. At viewpoint locations that 

do not have real image data, image-based rendering methods 

50 are used to generate geometric proxies that are combined 

with nearby image data, thus rendering synthetic views. In 

some embodiments, which image-based rendering algorithm 

is used to render the synthetic views depends on the density 

of the data that is collected, the camera geometry, charac55 teristics of the real scene, and so on. 

In one embodiment of the invention, a computer system 

provides image-based rendering of real property, the computer system including a user interface for visually presenting an image-based rendering of a real property to a user; 

60 and a processor to (i) obtain two or more photorealistic 

viewpoints from ground truth image data capture locations; 

(ii) combine and process two or more instances of ground 

truth image data to create a plurality of synthesized viewpoints; and (iii) visually present a viewpoint in a virtual 

65 model of the real property on the user interface, the virtual 

model including photorealistic viewpoints and synthesized 

viewpoints. 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 13 of 22
US 10,592,973 B 1 

3 4 

In one embodiment of the invention, a method obtains two 

or more photorealistic viewpoints of a real space from 

ground truth image data capture locations of the real space, 

generates a plurality of synthesized viewpoints by combining and processing two or more instances of the obtained 5 

ground truth image data, and presents, via a user interface, 

described in detail to avoid unnecessarily obscuring the 

description of the embodiments of the invention. 

Overview 

In some embodiments of the invention, a website implemented in a computer system allows users to navigate a 

virtual tour of a real property. Among its many uses 

described herein, embodiments of the website system provide a capability to tour a real property without physically 

visiting the property, to buy/sell/rent/lease-book the propa viewpoint in a virtual model of the real space, the virtual 

model including photorealistic viewpoints and synthesized 

viewpoints of the real space. 

Under an embodiment of the invention, a website system 10 

maintains a real estate web page. The real estate web page 

facilitates three-dimensional (3D) image-based rendering 

virtual tours of real properties through a unique user interface that provides multiple viewpoints and tour navigation 15 

tools. The website system facilitates virtual tours of real 

estate, such as homes, that are offered for sale via the 

website system. The website system can store various information about properties in which the user is interested and 

facilitate sharing of information with a real estate agent or 20 

other service provider (e.g., a local merchant). 

erty, to store and share portions of virtual tours, and to search 

for objects to purchase from merchants. In some embodiments, the system displays information on the web page but 

other delivery methods can be used (such as streaming video 

or email). 

Embodiments of the invention include numerous innovative informational, analytical, and collaborative tools. 

Some embodiments of the user interface provide valuable 

data that is aggregated from system user trends. Thus 

property sellers can see how many system users have viewed 

their virtual property, what areas of the virtual tour were 

most interesting to the viewers, and so on. On an individual 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 shows a block diagram of an embodiment of a 25 

computer system suitable for use with the disclosed invenlevel, this "backend" data can also be used for targeted 

advertising. For example, if a visitor to the virtual property 

is viewing the kitchen, the website system might show an 

advertisement for an appliance store from which the refrigerator in the virtual kitchen may be purchased. Similarly, the 

tions. 

FIG. 2 shows a block diagram of a system for providing 

an image-based rendering of a real property according to one 

embodiment of the invention. 

FIG. 3 shows a high level flow chart of the process of 

image-based rendering according to one embodiment of the 

invention. 

FIG. 4 shows a user interface for presenting image-based 

rendering or real property according to one embodiment of 

the invention. 

FIG. 5 shows a flow chart of a method of image-based 

rendering for real estate according to one embodiment of the 

invention. 

FIG. 6 shows a flow chart of a method of capturing and 

processing image data according to one embodiment of the 

invention. 

FIG. 7 shows a data capture map according to one 

embodiment of the invention. 

FIG. 8 shows a flow chart of a method of image-based 

rendering according to one embodiment of the invention. 

The headings provided herein are for convenience only 

and do not necessarily affect the scope or meaning of the 

claimed invention. 

system might show an advertisement for a furniture store 

that specializes in furniture in a style similar to that in which 

the virtual property is decorated. The website system may 

30 have metadata on objects in the 3D virtual model. The object 

metadata may be used to more closely match advertisers and 

advertisements to viewed content. For example, if a viewer 

looks at an object from several viewpoints, the type of the 

object ( e.g., cooktop) and its associated metadata ( e.g., 

35 Viking, 4-bumer, embodiments of the system can automatically recognize that a particular object having metadata, in 

this case the stovetop, that remains in the viewpoint can be 

used to trigger a targeted advertising system by correlating 

the object to an advertiser who sells the object or other 

40 products related to the object. Demographics and other data 

about the viewer may also be used in advertisement selection. The advertisements may be predetermined or an advertisement opportunity may be auctioned on-the-fly to a pool 

of advertisers who will bid for the opportunity to present an 

45 advertisement in the available advertisement location. 

In some embodiments, the system includes mechanisms 

for collaboration between various users of the system. 

Shared access to select user account information and a 

messaging system allow users to share their "favorites" 

50 folder with their real estate agents, leave comments and 

questions for home sellers, receive "suggested" property 

tours in the users "suggested" folder, and append comments 

or notes that will be visible to the user. 

In the drawings, the same reference numbers and acronyms identify elements or acts with the same or similar 

functionality for ease of understanding and convenience. To 

easily identify the discussion of any particular element or 

act, the most significant digit or digits in a reference number 

generally refers to the Figure number in which that element 55 

is first introduced (e.g., element 110 is first introduced and 

discussed with respect to FIG. 1). 

Example Computing Environment for Image-Based Rendering of Real Spaces 

FIG. 8 shows a block diagram of an exemplary embodiment of a system to implement the methods of image-based 

rendering for real estate and other real scenes disclosed 

DETAILED DESCRIPTION herein. A user may access the virtual model via a user 

The following description provides specific details for a 

thorough understanding of, and enabling description for, 

these embodiments of the invention. However, a person of 

ordinary skill in the art will understand that the invention 

may be practiced with many variations and these details do 

not list every possible variation. In some instances, well 

known structures and functions have not been shown or 

60 interface provided over the Internet 106 via a real estate 

webpage 110 or streamed media hosted on a remote server 

108. Alternatively, the user interface may be hosted locally 

on smartphone 102 or mobile computer 104, either as a 

standalone application or cached media. When the user 

65 selects a viewpoint location in the virtual model of a real 

property, a rendering engine, which may be hosted on the 

server computer 108, will retrieve processed image data 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 14 of 22
US 10,592,973 B 1 

5 

from database 112. The rendering engine will then render the 

requested viewpoint. In the embodiment shown, the server 

computer 108 will then provide requested 3D rendering to 

the user via real estate web page 110 and the user interface. 

FIG. 2 shows an exemplary computing environment 200 

for implementing various aspects of the disclosed inventions. The computing environment 200 includes a computer 

202, the computer 202 including a processing unit 204, a 

system memory 206 and a system bus 208. The system bus 

208 couples system components including, but not limited 

to, the system memory 206 to the processing unit 204. The 

processing unit 204 may be any of various commercially 

available processors. Dual microprocessors and other multiprocessor architectures may also be employed as the processing unit 204. 

The system bus 208 can be any of several types of bus 

structure that may further interconnect to a memory bus 

(with or without a memory controller), a peripheral bus, and 

a local bus using any of a variety of commercially available 

bus architectures. The system memory 206 includes read 

only memory (ROM) 210 and random access memory 

(RAM) 212. A basic input/output system (BIOS) is stored in 

a non-volatile memory 210 such as ROM, EPROM, or 

EEPROM. A BIOS contains the basic routines that help to 

transfer information between elements within the computer 

202, such as during start-up. 

The computer 202 further includes a hard disk drive 214. 

6 

a game port, a universal serial bus ("USB"), an IR interface, 

a wireless transceiver 258, etc. A monitor 220 or other type 

of display device is also connected to the system bus 208 via 

an interface, such as a video adapter 218. In addition to the 

5 display 220, a computer typically includes other peripheral 

output devices (not shown), such as speakers, printers, etc., 

that can present information to the user. 

As shown in FIG. 2, the computer 202 may operate in a 

networked environment using logical connections via wired 

10 and/or wireless communications to one or more remote 

computers, such as a remote computer(s) 248. The remote 

computer(s) 248 may be a workstation, a server computer, a 

router, a personal computer, portable computer, microprocessor-based entertainment appliance, a peer device or other 

15 common network node, and typically includes many or all of 

the elements described relative to the computer 202, 

although, for purposes of brevity, only a memory storage 

device 250 is illustrated. The logical connections depicted 

include a local area network (LAN) 252 and a wireless local 

20 area network (WLAN) 254. Such networking environments 

are commonplace in homes and businesses. The Internet can 

also be used to provide access to remote computer 248. 

When used in a LAN networking environment, the computer 202 is connected to the local network 252 through a 

25 wired or wireless communication network interface or 

adapter 256. The network adaptor 256 may facilitate wired 

or wireless communication to the LAN 252. When used in 

The hard disk drive 214 can be connected to the system bus a WLAN networking environment, the computer 202 typi208 by a hard disk drive interface 216. The removable 

storage drives (DVD drives, floppy drives, etc.) are not 30 

shown for clarity. However, the removable storage drives 

and their associated computer-readable media provide nonvolatile storage of data, data structures, and computerexecutable instructions for implementing the inventions 

described herein. For the computer 202, the drives and 35 

media accommodate the storage of information input by a 

user, or received from a remote computer, in a suitable 

digital format. Although the description of computer-readable media above refers to a hard disk, a removable magnetic disk, and a DVD, a person of ordinary skill in the art 40 

understands that other types of storage media which are 

readable by a computer, such as zip drives, magnetic cassettes, flash memory cards, digital video disks, cartridges, 

and the like, may also be used in the exemplary operating 

environment, and further that any such media may contain 45 

computer-executable instructions for performing the methods of the present invention. 

cally is connected to a communications server on the LAN, 

or has other means for establishing communications over the 

WLAN 254, such as the Internet. In a networked environment, program modules depicted relative to the computer 

202, or portions thereof, may be stored in the remote 

memory storage device 250. The network connections 

shown are exemplary and other means of establishing a 

communications link between the computers may be used. 

The computer 202 is operable to communicate with any 

other devices having wireless communication capability, 

e.g., a cell phone, a printer, desktop and/or portable computer, portable data assistant, and telephone. As discussed 

briefly above, suitable wireless technologies may include, 

but are not limited to, cellular, WLAN (e.g., IEEE 802.11), 

IEEE 802.16, IEEE 802.20, and Bluetooth. 

IEEE 802.11, also commonly known as "Wifi", is a 

wireless communication protocol that enables computers to 

send and receive data anywhere within the range of a base 

station. A WLAN can be used to connect computers to each 

other, to the Internet, and to wired networks (which may use 

IEEE 802.3 or Ethernet communication protocols). 

Software applications can be stored in the drives and 

RAM 212. These applications can include an operating 

system 230, one or more application programs 232, (e.g., 50 

web browsers and client applications, etc.) other program 

modules 234 (e.g., cookies, etc.) and program data 236. All 

Examples of Image-Based Rendering of Real Scenes 

FIG. 3 shows a high level flow chart 300 of the three 

primary steps in image-based rendering for real estate. In 

step 301, image data is captured. Image data may be 

captured by smartphones or other mobile devices, video 

or portions of the operating system, applications, modules, 

and/or data can also be cached in the RAM 212. 

Embodiments of the invention can be implemented with 

various commercially available operating systems or combinations of operating systems. 

A user can enter commands and information into the 

computer 202 through a keyboard 244 and a pointing device, 

such as a mouse 242. For example, the user might employ 

the mouse to navigate a virtual tour user interface, as 

described herein. Other input devices (not shown) may 

include a microphone, an IR remote control, a joystick, a 

game pad, similar devices. These and other input devices are 

often connected to the processing unit 204 through a serial 

port interface 240 that is coupled to the system bus 208, but 

may be connected by other interfaces, such as a parallel port, 

55 cameras, digital SLR cameras, specialized cameras (e.g., 

other cameras that take spherical panoramic images), etc. 

Additional geometry data about a real scene may also be 

captured from laser range scanners, infrared structure light, 

or other data collection. In step 302, the captured image data 

60 is processed to create panoramas, determine camera geometry, and 3D reconstruction algorithms are employed to 

generate dense representations of a 3D model, geometric 

proxies, parcel maps and floor plans. In step 303, one or 

more rendering algorithms are applied to the data and 3D 

65 model to render a 3D view of the real estate property. 

FIG. 4 shows an embodiment of a User Interface System 

400 for image-based rendering of real estate. Three different 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 15 of 22
US 10,592,973 B 1 

7 

user interface elements serve the dual purposes of informing 

the user of his location in the model and simultaneously 

enabling spatial navigation. These three elements are shown 

in the user interface (400) embodiment of FIG. 4, which 

would normally be contained within a browser window, 

within a framed client application, or as the entire screen 

during full screen mode. User Interface element 402 is the 

viewpoint within a virtual model generated by combining 

geometry and image data using Image-Based Rendering 

(IBR), thus creating a 3-dimensional (3D) view. UI element 

404 is a two-dimensional (2D) map overlay that displays the 

relative location in the virtual model of the current viewpoint 406 shown in UI element 402. UI element 408 is a text 

overlay that displays one or more labels associated with the 

user's location within the virtual model. 

FIG. 4 shows the three primary user interface elements of 

an embodiment of the navigation tool for image-based 

renderings of real estate. UI element 402 shows a rendering 

of the model from a specific viewpoint position and also 

serves as a means to navigate to adjacent positions and to 

change the viewing vector from a fixed position. Navigation 

can occur using various well known input/output (IO) 

devices, such as a keyboard, touchscreen, eye-tracking technology, gesture recognition technology, or a computer 

mouse. For densely sampled spherical panoramas, one 

example of navigation using UI element 402 with a mouse 

would be to click on the rendered view to translate in the XY 

plane to another panorama location in the model. Another 

example of navigation using UI element 402 would be to 

click and hold the mouse button, enabling rotation about the 

Z axis, thus "looking around" without translating in the XY 

plane. As the user navigates, the rendered viewpoint shown 

in UI element 402 changes in real time based on a new 

position and viewing vector associated with the new location. 

Many parts of the User Interface System 400 work 

together to create a unique user experience for touring a real 

estate environment over the Internet-in particular the 3D 

view rendered in UI element 402 warrants further discussion. In the 3D view of UI element 402, multiple IBR 

algorithms can be combined to create a user experience that 

overcomes the inherent tradeoff between photorealism and 

spatial navigation caused by using just one IBR algorithm 

exclusively. For example, dense spherical panoramas may 

be combined with the use of view dependent texture mapping (VDTM) during navigation as the user spatially translates and rotates the viewpoint with the virtual model. 

The capture process ( defining a spatial boundary, data 

sampling, and annotation) for an enhanced user experience 

involves a dense sampling of spherical panorama image data 

at multiple exposures (e.g., high definition rendering, HDR) 

and sampling of point cloud geometry from a hybrid camera 

and laser range scanning device. The processing pipeline 

( determining data sets, image calibration, image processing, 

3D reconstruction (3DR) and scene understanding) operates 

on the data output from the capture process. Image calibration involves determining panorama pose and adding depth 

to each pixel of the panorama. 

Image processing creates HDR spherical panoramas from 

input images, 3D reconstruction involves removing noise 

from point clouds, reconstructing a real estate environment's 

geometry to varying degrees of approximation, generating 

geometric proxies that describe the environment with simpler meta primitives, feature matching between spherical 

panoramas, positioning of spherical panorama data in 3D 

space, feature matching between the panorama and 3D 

8 

space, and computing view dependent texture maps for the 

geometry and/or geometric proxies. 

Rendering displays the processed data to an end-user via 

an IO device. During rendering, the user's position and 

5 navigation influence which elements of geometry and image 

data are combined for a given Image-Based Rendering 

algorithm at any possible location in or around the virtual 

model. Ground truth image data is the captured image 

associated with a particular capture location, and optionally 10 may include any metadata associated with the captured 

image, such as GPS coordinates, IR point clouds, etc. 

Ground truth data may also include labels (outside: backyard, or groundfloor: bathroom) which are ground truth in 

15 the sense that they are directly collected at the scene and are 

not synthetic approximations. For example, when a user is 

near a position where ground truth image data is captured, 

very little geometry is required to render the most photorealistic view of the model. At the exact position of capture, 

20 the use of an image primitive is the most photorealistic view 

possible. 

Composited image primitives such as a spherical panorama enable 2 DOF rotational navigation. When translating 

directly between two spherical panorama locations, other 

25 algorithms such as optical flow may provide more photorealistic warping during the rendering of predetermined translational pathways defined in the connectivity graph. When 

translating between other locations within the virtual model, 

the use ofVDTM over explicit geometric proxies combined 

30 with depth and feature matching between nearby panoramas 

during rendering provides a decrease in photorealism but 

enables fluid movement to any spatial location. 

In contrast to traditional texture maps, VDTMs compute 

new textures for different spatial positions, dynamically 

35 changing these as the user's spatial position changes. This 

dramatically reduces artifacts associated with appearance of 

objects. In various embodiments, rendering may use RGB 

spherical panoramas, point clouds, geometric proxies, view 

dependent texture mapping, and feature matching between 

40 the spherical panoramas and geometric proxies to create a 

fluid user experience involving multiple IBR algorithms that 

dynamically change based on the user's location, direction, 

and speed of movement within the model. A user can view 

ground truth image data or navigate to any other synthetic 

45 viewpoint in the same user interface, enjoying both photorealism and spatial navigation. 

FIG. 5 shows an embodiment of a computer-implemented 

image-based rendering (100) for providing a three-dimensional (3D) virtual model of a real property according to 

50 various aspects of the disclosed innovations. 

In step 502, the system creates or generates the spatial 

boundaries in a model of the property that define the land 

(perhaps a parcel outline), structure ( e.g., house, apartment, 

office, etc.), structure internals (e.g., bedrooms, kitchens, 

55 hallways, etc.) and/or air-space above the land (e.g., via 

quadcopter or pole-based data capture of aerial views). 

In step 504, RGB image data is captured from each of the 

defined spatial boundaries. The captured RGB data may be 

annotated, during or after capture, according to the spatial 

60 boundaries or other factors. The use of computational photography approaches such as High Dynamic Range (HDR) 

can greatly decrease the amount of work spent during the 

RGB capture stage because special lighting of the scene is 

not necessary. HDR captures the same image at multiple 

65 exposure levels and combines them with post-processing so 

that an appropriate exposure level is applied to each area in 

the captured scene. In some embodiments, the use of HDR 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 16 of 22
US 10,592,973 B 1 

9 

techniques allows for automatic adjustment for image exposure levels by selection of appropriate light levels for each 

pixel in the HDR images. 

10 

In step 516, the 3D data is used to generate a geometric 

proxy for the 3D model of the real property. 

In step 506, panorama images are created from the RGB 

image data. In an alternative embodiment, High Dynamic 5 

Range (HDR) panoramas can be constructed by processing 

multiple overlapping input images having varying exposures 

In step 518, the results are combined and a 3D scene that 

has photorealistic views from within a panorama capture 

location, as well as semi-photorealistic views from other 

locations using a geometric proxy and real image data 

(image-based rendering), are rendered to a viewer ( end 

user). In addition to the 3D views, 2D floor plans are shown 

to the user as well as descriptive spatial boundary labels (i.e., 

one dimensional or ID, such as a "list view" or a list of 

labels for different locations which map to the 3D model). In 

to create composite higher contrast images that are optimally 

exposed. 

In step 508, the camera geometry is determined by use of 10 

first pass feature detection within the panoramas in order to 

spatially calibrate the panoramas. The spatial boundary 

definitions may be used to group panoramas so that camera 

geometry is only found between line-of-site panoramas. A 15 

spherical panorama bundle adjustment heuristic can be used 

this way, the system can calculate and present a viewer with 

images of a real property in 3D, 2D, and ID. 

FIG. 6 provides a detailed block diagram of an embodiment of a method of processing image data and associated 

data according to one or more of the disclosed innovations. 

to group the panoramas. The bundle adjustment heuristic 

may be initialized with metadata such as GPS or wifi 

positioning coordinates associated with a sample capture 

location, gyroscopic or inertial data, or other spatial infor- 20 

In step 601, input images are collected. The input images 

can come from various sources such as mobile devices (e.g., 

smartphones), point-and-shoot cameras, GoPros on a rig, 

and specialty cameras systems such as LadyBug5 from Point 

mation. Grey. The input images can be shot with a tripod, or by 

In step 510, a second pass feature detection algorithm is 

applied to generate a dense representation of the 3D model 

geometry ( e.g., high level features such as planes, lines, 

floors, etc.) or individual 3D points which can create a dense 

point cloud. Note that any other geometric data collected 

from the scene such as with a laser range scanner, infrared 

based depth maps (e.g., such as from a Microsoft Kinect), or 

other manual approaches can increase the accuracy of the 

geometry but are not required in the disclosed system. Some 

embodiments of the system work with images and do not use 

other geometric data. 

In step 512, the geometry is converted into the correct 

geometric proxy based on the density of the data sampled 

during capture. The correct proxy is the one that can be 

combined with image data to create the most photorealistic 

synthetic viewpoint. The proxy is chosen from among 

various image-based rendering techniques such as viewdependent geometry, texture-mapped models, 3D warping, 

view-dependent texture, view morphing, view interpolation, 

mosaicking, concentric mosaics, light field, lumigraph, optical flow, stitching, rectification, color interpolation, blending, fall-off correction, etc., depending on tradeoffs between 

photorealism, desired amount of spatial navigation for the 

viewer (i.e., how much can the viewer vary the viewpoint), 

and the degree of data sampling. 

IBR algorithms generally differ based on a variety of 

factors, including, but not limited to, the type of geometric 

proxy employed in rendering; the density of sampling that 

takes place during data capture; the types of devices and 

sensors used in data capture; or the degree of spatial navigation enabled in the rendering or viewing experience. In 

general, more geometry enables more spatial navigation and 

less dense sampling. Less geometry and more image sampling enable higher degrees of photorealism but less spatial 

navigation. All IBR algorithms have a trade-off between 

photorealism, spatial navigation, and the degree of data 

sampling. Also, photorealism can be increased for synthetic 

views by use of other geometric data such as laser/infrared 

but such geometric data is not required by the disclosed 

innovations for suitable results. 

In step 514, the 3D data is used to generate a parcel map 

(e.g., the exterior and aerial views of the real estate) and 

floor plan. A 2D floor plan may be used to create a navigation-enabled map of the interior of the real estate. For 

example, a 2D floor plan of the rooms of a house can be 

automatically generated from the 3D data. 

holding the device in hand or on top of other devices such 

as drones, robots or other kinds of automated, semi-automated, or remote controlled devices. The output of such 

25 source devices is RGB information that may be stored on the 

local devices, streamed to nearby devices in real-time or in 

regular intervals, or transmitted to a remote storage device 

such as a server computer. The streaming of such RGB data 

can happen over WiFi or via wired communication charmels 

30 such as high-speed USB links. The input image data can be 

collected as singular shots, burst mode pictures, or as a 

video. The RGB input capture may be coordinated with 

capture of depth data (e.g., step 611) in time or location. 

The relative configuration between the locations of the 

35 RGB capture device and the depth capture device may be 

known in advance (rigid spatio-structural configuration) or it 

may be unconstrained (in which case their relative extrinsics 

will be approximated using computer vision techniques). In 

various embodiments, the input data may be captured by 

40 trained professional operators or by untrained consumers. 

Other data inputs can include GPS coordinates, indoor GPS 

coordinates, and approximate positions marked by a human 

operator on a floor plan. Additionally, in an alternative 

embodiment, a human operator can associate metadata such 

45 as tags with each image to represent the home, floor or room 

of the panorama ( e.g., 615). 

In step 602, the input images are processed using techniques such as white balancing, noise reduction, image 

stabilization, super-resolution, tone-mapping or other 

50 dynamic range compression techniques. Image processing 

can occur asynchronously or could be processed in realtime. In some embodiments it is possible to partially or fully 

process the data on the capture device. In other embodiments, the data may be uploaded/streamed to a remote server 

55 to be processed in order to take advantage of the server's 

greater processing power. 

In step 603, images whose extrinsics primarily differ by a 

rotational component may be stitched into a spherical panorama, a cylindrical panorama or a flat mosaic. Stitching 

60 these images together may be done using relative camera 

extrinsics, feature matching, graph cut algorithms (e.g., 

Boykov), image blending, panorama weaving, and other 

similar techniques. The stitching may be done before or after 

the image processing techniques described in step 402. In 

65 some scenarios, such as a homeowner capturing images with 

a smartphone, it may be difficult to detect if a set of images 

make a panorama. In those cases, the user could be explicitly 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 17 of 22
US 10,592,973 B 1 

11 

asked to capture images for a panorama. Alternatively, 

spatio-temporal coherence in the images may be used to 

detect panoramas. 

12 

may then be obtained by connecting panoramas that have 

strong feature matches. Algorithms such as kd-tree nearest 

neighbor search may be used to perform fast feature matching between images. Alternatively, brute force search may In step 610, embodiments of the system optionally may 

collect depth data that may be spatio-temporally coherent 

with the RGB data (for example, the depth sensors could be 

very close to the RGB sensors and could be sampled at 

around the same time when the RGB sensors are sampled). 

The depth data is in addition to RGB data provided by 

camera devices and may come from different sources such 

5 be used, although generally less effectively. The panorama 

neighborhood graph may be also be constructed with the 

help of additional metadata information (e.g., room labels, 

GPS coordinates, etc) associated with the panoramas or a 

manually connected panorama graph. A weight is typically 10 assigned to each edge to indicate the strength of the feature 

match between the edges of the panorama pairs. The weight 

can be computed by estimating the reprojection error after 

the relative extrinsics have been estimated. 

as Infrared (IR), LASER, or structured light. The depth data 

may be coupled with RGB data in either an open or 

proprietary format. Other data inputs associated with the 

image data can include GPS coordinates, indoor GPS coordinates, and approximate positions marked by a human 15 

operator on a floor plan of the property. 

In step 606, the panorama connectivity graph (neighborhood graph) constructed in step 605 may be used to estimate 

the relative extrinsics between panoramas. This information 

can be used to reduce the jarring effects when transitioning 

between these panoramas in the Image Based Rendering 

20 module. 

In step 611, RGB data may be registered with depth data 

(sometimes herein referred to as RGB+D data) by using 

different heuristics such as using relative directional differences in the sensors (assuming the relative positions of the 

sensors is insignificant) or a per-pixel registration of the 

RGB and depth sensor data can be achieved by warping the 

depth image to the perspective of the RGB sensor image and 

computing per-pixel depth in the RGB space after processing the warped depths. In some embodiments, registering 25 

RGB data with depth data may be used to interpret a pixel 

location from a particular image as a 3D ray or vector for 

later use in mapping texture onto a 2-D or 3-D polygon mesh 

or a point cloud. 

In step 612, a collection of RGB+D images may be 30 

registered together using an algorithm such as Iterative 

Closest Point (ICP) or other similar algorithms for reducing 

the difference between two clouds of points or geometric 

alignment of 3D models. Factors such as relative proximity 

of the depth images may be used to initialize the ICP 35 

algorithms. A globally consistent point may be obtained by 

pamng spatio-temporally neighboring depth images 

together and progressively merging locally registered point 

clouds into successively larger and more global point clouds. 

A distributed ICP algorithm may be employed to operate on 40 

local point clouds independently with successively larger 

point clouds being processed by larger individual nodes. 

In step 617, an optional preview mode may be used to 

provide preliminary feedback regarding the quality of data 

and the quality of the eventual user experience. The preview 45 

feature may range from previewing raw sampled RGB and 

depth data, to previewing semi-processed images ( eg., 

stitched panoramas, white balanced images, etc.) to previewing part or all of reconstructed imagery and geometry. 

In step 604, the panorama image results of step 603 are 50 

further processed for feature mapping. Feature matching 

between panoramas is an important way to understand the 

relative spatial relationships between panoramas. Feature 

detection may be done using standard feature detectors such 

as SIFT, Harris corner detector, MSER detector, Carmy edge 55 

detector, or SURF detector. In indoor environments, feature 

detection is also useful to find higher level features such as 

rectangles. The features may be matched using nearest 

neighbor, approximate nearest neighbor algorithms, or other 

non-parametric methods used for classification and regres- 60 

sion. Additional constraints such as ratio tests or scale 

invariant feature tests may be used to improve the accuracy 

of the feature matches. 

In step 607, a sparming tree of the panorama neighborhood graph may be useful to simplify the initial extrinsics 

estimation because the panorama connectivity graph is a 

densely connected graph. The spanning tree can optionally 

be constructed so that it spans strongly connected clusters of 

panoramas (e.g., all the panoramas in the same room) as 

opposed to sparming individual panoramas. A root is chosen 

of this panorama spanning tree and the global extrinsic is 

propagated along the graph by using the relative extrinsics 

computed between the 2 panoramas of each edge. This 

global extrinsic may be used for the initial input to the 

panorama bundle adjustment process. In some embodiments, the initial estimates of the extrinsics may be 

improved by constraining the position and orientation of the 

panoramas using spanning tree limitations. 

In step 608, bundle adjustment is the process of simultaneously estimating the 3D positions of the points corresponding to the matched features and extrinsics of the 

panoramas. Bundle adjustment attempts to minimize reprojection error between the image locations of observed and 

predicted image points ( e.g., minimizing deviation from true 

image projections). A good outcome of bundle adjustment 

requires good initial estimates of the 3D point positions and 

the camera extrinsics. The likelihood of a good outcome is 

improved by detecting a feature in as many panoramas as 

possible. In other words, the higher the average number of 

panoramas in which a feature is detected, the higher the 

accuracy of the computed output. In one embodiment, a 

bundle adjustment formulation is implemented in which 

bundle adjustment is initially done on small clusters of 

strongly connected panoramas. This technique usually 

results in good local estimates. The small cluster bundle 

adjustments may then be merged one by one. 

In step 609, computed global panorama extrinsics result 

from the bundle adjustment of step 608. 

In step 613, the computed globally consistent 3D point 

cloud is derived from ICP-based registration of depth sensored data in step 612 and/or an output of bundle adjustment 

step 608. 

In step 614, higher level geometric proxies are fit to the 

point cloud. These proxies may include line segments (ID), 

curves (ID), planes (2D), rectangles (2D), triangles (2D), 

and 3D voxel based volumetric representations. Several 

In step 605, a panorama neighborhood graph may be techniques such as noise elimination, Manhattan World 

created. The feature matching information may be run on 

different pairs of panoramas to ascertain the spatial proximity of the panoramas. A proximity graph of the panoramas 

65 assumptions, RANSAC-based plane fitting, and Poisson 

surface reconstruction can be used to derive higher level 

primitives representing the scene. Meta level representations 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 18 of 22
US 10,592,973 B 1 

13 

of the scene such as the floor plan can be derived by 

simplification of these primitives. 

In step 615, metadata may be optionally associated with 

images or portions of images to identify objects ( eg., oven, 

fridge, etc.), to explain a space (eg., ways to use a physical 

space, etc.), to provide location information, etc. In one 

alternative embodiment, object metadata may be automatically determined by comparing an object's 3D image with a 

database of known images. A match ( e.g, via comparison of 

point cloud data, image features, etc.) to a known image will 

result in identification of the object and the appropriate 

metadata may be imported from the database and associated 

with the newly identified object. As additional metadata, the 

creator of the virtual model may identify object attributes 

that are useful for advertisers, such as model numbers, color, 

physical size of opening or object, similar styles, etc. Knowing the physical size of an object or the size of the opening 

into which it fits allows for the advertisement of potential 

substitutes for the object. For example, knowing the size of 

the opening available for a refrigerator would allow the 

viewer to browse potential substitutes of the correct size 

from an advertiser. 

In step 616, the stitched panoramas, the geometric information, raw captured input data, and metadata are then 

combined to achieve photo-realistic rendering using imagebased rendering techniques. 

FIG. 7 shows an example of a panorama capture map 700 

having of a distribution of captured spherical panoramas on 

a parcel 701 having a house 702, where each number 1-21 

represents a Data Sampling location ( e.g., a location where 

a spherical panorama photo was taken). Note that different 

sampling densities in different spatial boundaries, such as 

dense sampling indoors 704 and sparse sampling outdoors 

706. Details associated with the floor plan and topography 

are omitted. Two arrows shown between capture location 2 

and 14 and 11 and 20 represent doors that connect the spatial 

boundaries in a connectivity graph. In later rendering stages, 

users will be able to move between the indoor and outdoor 

areas based on this connectivity graph. Additional details 

associated with the topography and the interior floor plan are 

omitted for clarity as are the actual density of the spherical 

panoramas. 

FIG. 8 shows a flowchart of an exemplary embodiment of 

an image-based rendering method. 

In step 801, a user may request different kinds of views as 

a user navigates a sampled RGB+D environment (e.g., a 

virtual model of house). In some instances, the user may 

request a view from the same location at which image data 

was captured. In other instances, the user may request a view 

from a location at which no image data was captured. If the 

user is navigating a sampled video path, for example, then 

the user may request a view from a location where a video 

frame was not captured. Or the user may want to jump off 

from the sampled path. 

When a user is navigating panoramas, another type of 

view that a user may request is a transition between panoramas that is indicative of the physical experience of 

transitioning between the panorama capture locations in the 

real world. For example, the user may want to virtually 

move from the location where a panorama was captured to 

a nearby location where a different panorama was captured. 

Alternatively, the user may want to get an approximate view 

that involves significantly stepping away from the locations 

where the image data was captured. A smooth visual transition between the virtual locations avoids the visual dissonance of an abrupt image change. 

14 

At decision point 802, the rendering system evaluates the 

data at its disposal to aid the rendering of a requested 

viewpoint. If the viewpoint has been sampled as panorama, 

an image or as a video, then the rendering system can use the 

5 sampled raw and/or processed sensor-captured data to render the requested viewpoint in step 503. However, the 

requested viewpoint may be relatively far away from any 

sampled viewpoint. This is quite possible in real estate 

environments where the area can be very large (e.g., large 

10 homes, ranches, event spaces), acquisition can be difficult 

(e.g., view of the home from the top, heavy obstacles to 

accessing a region) or delicate ( e.g., water ways, soft greenery, etc.). Such spaces may prohibit a denser image sampling 

of the region and require synthesis of novel viewpoints, step 

15 804, at locations from which no image data was directly 

captured. As discussed in the explanation of FIG. 6, step 

512, choosing an algorithm for novel viewpoint synthesis is 

typically a function of tradeoffs between photorealism, the 

desired amount of the quality and density of the available 

20 data, desired amount of spatial navigation for the viewer 

(i.e., how much can the viewer vary the viewpoint), and the 

degree of data sampling. 

In step 807, light field rendering involves rendering a 

novel viewpoint by sampling the space of lights rays in and 

25 around the viewpoint's vicinity and reconstructing the light 

field at the requested viewpoint. If the sampling around the 

requested viewpoint is dense enough or the scene is not 

complicated enough (e.g., most of the complex geometry is 

very far away or there is very little RGB+D complexity in 

30 the vicinity of the requested view point) then the light field 

at the novel view point can be approximated by: 1) warping 

the RGB+D of the nearby sampled images to the new view 

point, 2) projecting the approximate depth map at the novel 

view point by synthetic rendering of the reconstructed 

35 geometry, 3) Using feature matching information to register 

the warped and synthetic geometry with respect to each 

other at the novel view point, 4) Run time rendering is then 

done using a warping, blending, -based or machine-learningbased approach to predict the object (and hence the RGB+D 

40 correspondence) at every pixel of the novel viewpoint. This 

requires an automated processing of the captured data to 

analyze if it lends itself to light field rendering. A 2D ( or 3 D) 

map is determined which identifies the areas where light 

field rendering is permissible. This map also stores the 

45 contributing sampled data at points in the permissible area. 

During render time, the sampled data is then used to render 

the novel view point. If the rendering is built on machinelearning based approaches, it may not be feasible to render 

in real time using a simple computing device. In such 

50 situations an oflline or distributed rendering infrastructure is 

leveraged. 

In step 805, view dependent texture mapping is a way to 

synthesize novel views by mapping different pre-captured/ 

synthesized texture maps on the same approximate geom55 etry. The resulting renderings tend to be a better viewpoint 

approximation. A 2D (or 3D) map is determined which 

identifies the areas where light field rendering is permissible. 

In each such region, the visible geometry and the permissible view-dependent texture maps for each polygonal ele60 ment of the geometry are computed. At render-time, the right 

texture map to use is determined for each polygon depending on the rendering viewpoint. 

In step 806, pure synthetic rendering from geometric 

proxies involves rendering captured/reconstructed geometry 

65 along with the computed texture maps. 

In step 808, the final rendering chosen for novel view 

synthesis depends on the situational accuracy of the various 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 19 of 22
US 10,592,973 B 1 

15 

rendering algorithms. The accuracy of rendering at a given 

location can be estimated using techniques such as reprojection errors, average texture warp distortions, average 

distance to sampled view points, etc. Depending on the 

estimated accuracy of various novel viewpoint synthesis 5 

algorithms, the estimation transitional speed of the viewer 

and the clarity of rendering expected by the user, the final 

rendering algorithm may be a weighted blend of two or more 

of the results of the various novel viewpoint synthesis 

algorithms from steps 805, 806 and 807. 10 

Aspects of the invention described above may be stored or 

distributed on computer-readable media, including magnetic 

and optically readable and removable computer discs, as 

well as distributed electronically over the Internet or over 15 

other networks (including wireless networks). Those skilled 

in the relevant art will recognize that portions or embodiments of the invention may also reside in a fixed element of 

a communication network such as a server or database, 

while corresponding portions may reside on a mobile com- 20 

munication device, such as a laptop computer, Personal 

Digital Assistant ("PDA"), or mobile phone. Data structures 

and transmission of data particular to aspects of the invention are also encompassed within the scope of the invention. 

In accordance with the practices of persons skilled in the 25 

art of computer programming, embodiments of the invention 

are described with reference to acts and operations that are 

performed by computer systems. Such computer-executed 

acts and operations may be performed by an operating 

system ( e.g., Microsoft Windows, Linux, Apple iOS, 30 

Android) or an application program. The acts and operations 

include the manipulation by the CPU of electrical signals 

representing data bits and the maintenance of data bits at 

memory locations to operate the computer systems and 

process signals. The memory locations where data bits are 35 

maintained are physical locations that have particular electrical, magnetic, or optical properties corresponding to the 

data bits. Although databases are shown as separate physical/logical entities for clarity, they may be combined and/or 

aggregated to suit the physical/logical architecture of the 40 

system in which they are used. 

Unless the context clearly requires otherwise, throughout 

the description and the claims, the words "comprise," "comprising," and the like are to be construed in an inclusive 

sense as opposed to an exclusive or exhaustive sense; that is 45 

to say, in the sense of"including, but not limited to." Words 

using the singular or plural number also include the plural or 

singular number respectively. Additionally, the words 

"herein," "above," "below" and words of similar import, 

when used in this application, shall refer to this application 50 

as a whole and not to any particular portions of this 

application. When the claims use the word "or" in reference 

to a list of two or more items, that word covers all of the 

following interpretations of the word: any of the items in the 

list, all of the items in the list and any combination of the 55 

items in the list. 

The above detailed descriptions of embodiments of the 

invention are not intended to be exhaustive or to limit the 

invention to the precise form disclosed above. While specific 

embodiments of, and examples for, the invention are 60 

described above for illustrative purposes, various equivalent 

modifications are possible within the scope of the invention, 

as those skilled in the relevant art will recognize. For 

example, while steps are presented in a given order, alternative embodiments may perform routines having steps in a 65 

different order. The teachings of the invention provided 

herein can be applied to other systems, not necessarily the 

16 

embodiments described herein. These and other changes can 

be made to the invention in light of the detailed description. 

These and other changes can be made to the invention in 

light of the above detailed description. In general, the terms 

used in the following claims should not be construed to be 

limited to the specific embodiments disclosed in the specification, unless the above detailed description explicitly 

defines such terms. Accordingly, the actual scope of the 

invention encompasses the disclosed embodiments and all 

equivalent ways of practicing or implementing the invention 

under the claims. 

In view of the many possible embodiments to which the 

principles of this invention may be applied, it should be 

recognized that the detailed embodiments are illustrative 

only and should not be taken as limiting the scope of the 

invention. Thus, we claim as our invention all such embodiments as may come within the scope and spirit of the 

following claims and equivalents thereto. 

What is claimed is: 

1. A computer-readable memory medium containing program commands for controlling a computer processor, when 

executed, to provide image-based rendering of a real property, by performing a method comprising: 

obtaining two or more views from capture locations in a 

plurality of spaces of the real property using ground 

truth image data, the ground truth image data including 

images of the real property captured from the capture 

locations; 

defining a plurality of spatial boundaries of the plurality 

of spaces of the real property, each spatial boundary 

delineating a volume of one or more of the plurality of 

spaces, at least one of the plurality of spatial boundaries 

defining a parcel outline of the real property, and at 

least one of the plurality of spatial boundaries defining 

a space located within the parcel outline; 

annotating the ground truth image data by annotating at 

least some of the captured images of the real property 

with information indicating a capture location of a 

corresponding image relative to a corresponding spatial 

boundary, each capture location identifying a location 

within the corresponding spatial boundary defined 

space or identifying the space delineated by the corresponding spatial boundary; 

combining and processing two or more instances of the 

ground truth image data to create a plurality of synthesized views, wherein the synthesized views comprise 

one or more of composites, transitions, or projections 

derived from processing ground truth image data, and 

wherein the synthesized views are associated with 

corresponding spaces of the plurality of spaces of the 

real property; 

generating and rendering a virtual model of a current 

space within the plurality of spaces of the real property 

from a perspective of a current view, the current view 

being one of the plurality of views obtained using 

ground truth image data or one of the plurality of 

synthesized views derived from processing ground 

truth data and associated with the current space; 

identifying the current space in the real property where 

the current view is located using the capture locations 

indicated by the annotated ground truth image data; 

visually presenting, on the user interface, at least a portion 

of the virtual model, a map user interface element, and 

a text user interface element, the map user interface 

element indicating a position of the current view in the 

current space, the text user interface element including 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 20 of 22
US 10,592,973 B 1 

17 a label identifying the current space in the real property 

within which the current view is located; 

18 

11. The computer-readable memory medium of claim 1 

wherein the image-based rendering is performed using 

image-based rendering techniques comprising: techniques 

for rendering with no geometry, techniques for rendering 

wherein the portion of the virtual model, the map user 

interface element, and the text user interface element 

are functionally linked based on the position of the 

current view in the real property. 

5 with implicit geometry, or techniques for rendering with 

explicit geometry. 

2. The computer-readable memory medium of claim 1 

wherein the label identifying the current space in the real 

property is a floor designation, and wherein the annotating 

at least some of the captured images of the real property with 10 

information indicating a capture location of a corresponding 

image relative to a corresponding spatial boundary further 

comprises determining one or more spatial boundaries associated with the floor designation identifying the current 15 

space. 

3. The computer-readable memory medium of claim 1 

wherein a plurality of the visually presented at least a portion 

of the virtual model, the map user interface element, and the 

text user interface element are presented to the user at the 20 

same time. 

4. The computer-readable memory medium of claim 1 

wherein the map user interface element is a 2D projection of 

the rendered virtual model. 

5. The computer-readable memory medium of claim 1 25 

wherein the method further comprises: 

presenting a navigation interface that facilitates virtual 

movement to different positions in the real property 

using at least one of a visually presented portion of the 

virtual model, a map user interface element, or a text 30 

user interface element. 

12. The computer-readable memory medium of claim 1 

wherein the current space is a room in a building of the real 

property. 

13. The computer-readable memory medium of claim 1 

wherein the method further comprises: 

generating the map user interface element and the text 

user interface element using the annotations associated 

with the ground truth image data; and 

displaying an advertisement for an object in the at least 

the portion of the virtual model presented on the user 

interface, based upon a type of the object. 

14. The computer-readable memory medium of claim 1, 

wherein the text user interface element further includes a 

second label indicating a location of a different space of 

the plurality of spaces of the real property; and 

wherein the method further comprises changing the current view to a different view upon receiving a user input 

selecting the second label, the different view being 

positioned in the different space of the plurality of 

spaces of the real property. 

15. The computer-readable memory medium of claim 1 

wherein the method further comprises: 

receiving a request to schedule an in-person tour of the 

real property. 

16. The computer-readable memory medium of claim 1 

wherein the method further comprises: 

facilitating a transaction regarding the real property. 

17. The computer-readable memory medium of claim 16 

6. The computer-readable memory medium of claim 1 

wherein the combining and processing two or more 

instances of the ground truth image data to create the 

plurality of synthesized views further comprises: 

constructing one or more point clouds using the ground 

truth image data; 

generating one or more geometric proxies from the constructed point clouds, each geometric proxy comprising 

a model of geometric data with registered captured 

images or texture maps at associated points in the 

model of geometric data; and 

35 wherein the transaction further comprises one or more of: 

scheduling a tour, initiating messaging, initiating audio 

communication, customer relationship management, transaction document signing or tracking, on-line negotiations, 

offer submissions, inspection support, financing support, 

40 real-estate transaction workflows, data tracking, or reporting 

functions. 

rendering the at least some of the plurality of synthesized 

views from the generated geometric proxies. 

7. The computer-readable memory medium of claim 6 

wherein the generating and rendering the virtual model of 

the current space using at least one of the plurality of 

synthesized views is rendered from the generated one or 

more geometric proxies. 

18. A computer-readable memory medium containing 

program commands for controlling a computer processor, 

when executed, to provide image-based rendering of a real 

45 property, by performing a method comprising: 

8. The computer-readable memory medium of claim 6 50 

wherein the method further comprises: 

capturing geometric data collected from one or more 

sensors, and wherein the one or more point clouds are 

constructed from a combination of ground truth image 

data and geometric data, and the one or more generated 55 

geometric proxies utilize a combination of ground truth 

image data and geometric data. 

9. The computer-readable memory medium of claim 1 

wherein the method further comprises: 

creating a synthesized view that is a spherical panorama 60 

of captured ground truth image data, and wherein the 

rendered virtual model of the current space renders the 

spherical panorama. 

10. The computer-readable memory medium of claim 1 

wherein the synthetic views enable rotation of a presented 65 

portion of the virtual model about a vertical axis and a 

horizontal axis. 

obtaining two or more views from capture locations in a 

plurality of spaces of the real property using ground 

truth image data, the ground truth image data including 

images of the real property captured from the capture 

locations; 

defining a plurality of spatial boundaries of the plurality 

of spaces of the real property, each spatial boundary 

delineating a volume of one or more of the plurality of 

spaces, at least one of the plurality of spatial boundaries 

defining a parcel outline of the real property, and at 

least one of the plurality of spatial boundaries defining 

a space located within the parcel outline; 

annotating the ground truth image data by annotating at 

least some of the captured images of the real property 

with information indicating a capture location of a 

corresponding image relative to a corresponding spatial 

boundary, each capture location identifying a location 

within the corresponding spatial boundary defined 

space or identifying the space delineated by the corresponding spatial boundary; 

combining and processing two or more instances of the 

ground truth image data to create a plurality of syntheCase 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 21 of 22
US 10,592,973 B 1 

19 

sized views, wherein the synthesized views comprise 

one or more composites, transitions, or projections 

derived from processing ground truth image data, and 

wherein the synthesized views are associated with 

corresponding spaces of the plurality of spaces of the 5 

real property by: 

constructing one or more point clouds using the ground 

truth image data; and 

generating one or more geometric proxies from the 

constructed point clouds, each geometric proxy com- 10 prising a model of geometric data with registered 

captured images or texture maps at associated points 

in the model of geometric data; and 

generating and rendering a virtual model of a current 

space within the plurality of spaces of the real property 

from a perspective of a current view, the current view 15 

associated with the current space and being one of the 

plurality of views obtained using ground truth image 

data or one of the plurality of synthesized views 

derived from processing ground truth data; 

identifying the current space in the real property where 20 

the current view is located using the capture locations 

indicated by the annotated ground truth image data; 

visually presenting a portion of the rendered virtual model 

20 

determining a second panorama located nearby the 

second location using a point in the geometric proxy 

that corresponds to the second location; and 

visually presenting a transition to the determined second panorama resulting in visual presentation of the 

second panorama. 

19. The computer-readable memory medium of claim 18 

wherein the second panorama is not adjacent to the first 

panorama in the virtual model. 

20. The computer-readable memory medium of claim 18 

wherein the first and second panoramas are spherical panoramas. 

21. The computer-readable memory medium of claim 18 

wherein the transition is performed using one or more image 

based rendering techniques for rendering with no geometry, 

rendering with implicit geometry, or rendering with explicit 

geometry. 

22. The computer-readable memory medium of claim 18 

wherein the user indication to navigate to the second location is selection of a label corresponding to one of the 

defined spatial boundaries of the plurality of spaces of the 

real property. of the current space in a user interface as a navigable 

first panorama; and 25 

23. The computer-readable memory medium of claim 22 

wherein the label corresponds to an indoor area, an outdoor 

area, a floor, or a room of the property. 

in response to receiving a user indication to navigate to a 

second location within the plurality of spaces of the real 

property not contiguous to the current space, * * * * * 

Case 6:20-cv-00376-ADA Document 29-5 Filed 11/05/20 Page 22 of 22