Office Action 1: Non-Final Rejection (2024-01-05)
Rejection types: 102 - Lack of Novelty, 103 - Obviousness

CLAIMS:
WHAT IS CLAIMED IS:

1\. A computer-implemented method in which one or more processing devices perform operations comprising:
storing a three-dimensional (3D) virtual object that corresponds to a real-world object, the 3D virtual object including a superimposition of an image area showing a portion of the real-world object on a face of a 3D shape;
storing a virtual reset that includes information about the 3D virtual object and a position of the 3D virtual object in the virtual reset; and
presenting the virtual reset, the presentation showing the 3D virtual object at the position.

2\. The computer-implemented method of claim 1, further comprising:
generating the virtual object, wherein generating the virtual object comprises:
receiving, via a user interface of a device, a selection of the 3D shape that includes a plurality of faces including the face;
receiving an input indicating properties for the 3D shape;
receiving an image showing a portion of a real-world object;
determining an area of the image that corresponds to the portion of the real-world object;
associating, in the virtual object, the area of the image with a face of the plurality of faces of the 3D shape; and associating, in the virtual object or in metadata of the virtual object, the 12 properties with the virtual object.

3\. The computer-implemented method of claim 2, wherein associating, in the virtual object, the area of the image with the face of the plurality of faces of the 3D shape comprises:
presenting, via the user interface, the image imposed on the face, wherein boundaries of the portion of the real-world object in the image do not correspond with boundaries of the face;
providing, via the user interface, an editing interface object selectable to enable editing of the image; and
editing the image, responsive to receiving inputs to the editing object, so that the boundaries of the portion of the real-world object in the image correspond to the boundaries of the face.

4\. The computer-implemented method of claim 2, wherein the properties comprise one or more of dimensions or a weight.

5\. The computer-implemented method of claim 2, further comprising:
presenting, via the user interface, a set of three-dimensional shapes including the three-dimensional shape and one or more other three-dimensional shapes.

6\. The computer-implemented method of claim 5, wherein the set of three- dimensional shapes include one or more of a cube, a rectangular prism, a sphere, a spheroid, a cylinder, a cone, or a pyramid.

7\. The computer-implemented method of claim 1, further comprising generating the 3D virtual reset, wherein generating the 3D virtual reset comprises:
presenting, at a user interface, the 3D virtual object in the 3D virtual reset at a first position;
receiving, via the user interface, an edit to the 3D virtual object in the 3D virtual reset;
updating the presentation of the 3D virtual reset by showing the edit; and
storing the 3D virtual reset by including, in the 3D virtual reset, information about 9 the 3D virtual object and information about the edit.

8\. The computer-implemented method of claim 7, wherein the information about the 3D virtual object comprises the 3D virtual object or a link to the 3D virtual object.

9\. The computer-implemented method of claim 7, wherein the edit comprises one or more of a change in position of the 3D virtual object in the 3D virtual reset from the first position to a second position or a rotation of the 3D virtual object.

10\. The computer-implemented method of claim 7, wherein the edit comprises a change to the 3D virtual object comprising one or more of changes to images of one or more faces of the 3D virtual object or a resizing of the 3D virtual object.

11\. The computer-implemented method of claim 7, further comprising:
storing the edited 3D virtual object at a storage location in a data storage unit, wherein the information about the 3D virtual object and the information about the edit include the edited 3D virtual object or a link to the storage location of the edited 3D virtual object.

12\. The computer-implemented method of claim 7, further comprising:
storing the edit at a storage location in a data storage unit, wherein the information about the 3D virtual object includes the 3D virtual object or a link to the 3D virtual object and the information about the edit includes the edit or a link to the storage location of the edit.

13\. The computer-implemented method of claim 1, further comprising:
responsive to receiving an input in the augmented reality environment, moving the virtual reset within the augmented reality environment so that a location within the augmented reality environment of the virtual reset corresponds to a physical location in a physical environment of a physical reset to be assembled.

14\. The computer-implemented method of claim 1, wherein presenting the virtual reset comprises presenting the virtual rest in a virtual reality environment.

15\. The computer-implemented method of claim 1, wherein presenting the 2 virtual reset comprises presenting the virtual rest in an augmented reality environment.

16\. A system comprising:
a processor; and
a non-transitory computer readable medium storing computer-readable program instructions that, when executed by the processor, cause the system to:
storing a three-dimensional (3D) virtual object that corresponds to a real- world object, the 3D virtual object including a superimposition of an image area showing a portion of the real-world object on a face of a 3D shape;
storing a virtual reset that includes information about the 3D virtual object and a position of the 3D virtual object in the virtual reset; and
presenting the virtual reset, the presentation showing the 3D virtual object at the position.

17\. The system of claim 16, the operations further comprising:
generating the virtual object, wherein generating the virtual object comprises:
receiving, via a user interface of a device, a selection of the 3D shape that includes a plurality of faces including the face;
receiving an input indicating properties for the 3D shape;
receiving an image showing a portion of a real-world object;
determining an area of the image that corresponds to the portion of the real-world object;
associating, in the virtual object, the area of the image with a face of the plurality of faces of the 3D shape; and associating, in the virtual object or in metadata of the virtual object, the 12 properties with the virtual object.
18 The system of claim 16, the operations further comprising generating the 3D virtual reset, wherein generating the 3D virtual reset comprises:
presenting, at a user interface, the 3D virtual object in the 3D virtual reset at a first position;
receiving, via the user interface, an edit to the 3D virtual object in the 3D virtual reset;
updating the presentation of the 3D virtual reset by showing the edit; and
storing the 3D virtual reset by including, in the 3D virtual reset, information about the 3D virtual object and information about the edit.

19\. The system of claim 16 the operations further comprising:
responsive to receiving an input in the augmented reality environment, moving the virtual reset within the augmented reality environment so that a location within the augmented reality environment of the virtual reset corresponds to a physical location in a physical environment of a physical reset to be assembled.

20\. A non-transitory computer-readable medium having program code that is stored thereon, the program code executable by one or more processing devices for performing operations comprising:
storing a three-dimensional (3D) virtual object that corresponds to a real-world object, the 3D virtual object including a superimposition of an image area showing a portion of the real-world object on a face of a 3D shape;
storing a virtual reset that includes information about the 3D virtual object and a position of the 3D virtual object in the virtual reset; and
presenting the virtual reset, the presentation showing the 3D virtual object at the position. Page 40 of 45 Page 41 of 45 Page 42 of 45 Page 43 of 45 Page 44 of 45 1 2 3 4 5 6 7 8 9 10 11

OFFICE ACTION:
Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
Claims 1-13, 15-18, and 20 are rejected under 35 U.S.C. 102a1 as being anticipated by Geisner et al., US 2013/0083064 A1, hereinafter Geisner.
Claim 1.
A computer-implemented method in which one or more processing devices perform operations comprising:
storing a three-dimensional (3D) virtual object that corresponds to a real-world object, the 3D virtual object including a superimposition of an image area showing a portion of the real-world object on a face of a 3D shape (see [0035] discloses personal A/V apparatus 8 includes an optical see-through, augmented reality display device as a near-eye, augmented reality display device 2 in communication with a companion processing module 4 via a wire 6 in this example or wirelessly in other examples. In this embodiment, head mounted display device 2 is in the shape of eyeglasses in a frame 115, with a display optical system 14 for each eye in which image data is projected into a user's eye to generate a display of the image data while a user also sees through the display optical systems 14 for an actual direct view of the real world. See [0006-0007] for storing the 3D object, also see [0051]);
storing a virtual reset that includes information about the 3D virtual object and a position of the 3D virtual object in the virtual reset (see [0092] discloses the virtual data engine 195 performs translation, rotation, and scaling operations for display of the virtual data at the correct size and perspective. A virtual data position may be dependent upon, a position of a corresponding object, real or virtual, to which it is registered. The virtual data engine 195 can update the scene mapping engine about the positions of the virtual objects processed.); and
presenting the virtual reset, the presentation showing the 3D virtual object at the position (see [0041], [0051], [0127] and [0092]).
Claim 2.
The computer-implemented method of claim 1, further comprising:
generating the virtual object, wherein generating the virtual object comprises: receiving, via a user interface of a device, a selection of the 3D shape that includes a plurality of faces including the face (see [0070] discloses 0070] The scene mapping engine 306 is first described. A 3D mapping of the display field of view of the augmented reality display can be determined by the scene mapping engine 306 based on captured image data, and captured depth data if available, for the display field of view. The 3D mapping includes 3D space positions or position volumes for objects. A 3D space is a volume of space occupied by the object. Depending on the precision desired, the 3D space can match the 3D shape of the object or be a less precise volume like a bounding shape like a bounding box or bounding ellipse around an object. A 3D space position represents position coordinates for the volume or 3D space. In other words the 3D space position identifies how much space an object occupies and where in the display field of view that occupied space is.);
receiving an input indicating properties for the 3D shape (see [0100] FIG. 3C illustrates some examples of data fields in an object physical properties data set 320N stored for a real object, which includes data values detected or otherwise determined based on captured data of the real object, or for a specific virtual object. Also see [0070], [0079] and [0099]);
receiving an image showing a portion of a real-world object (see [0041], [0051] and [0092]);
determining an area of the image that corresponds to the portion of the real-world object (see [0079], [0092] and [0099]);
associating, in the virtual object, the area of the image with a face of the plurality of faces of the 3D shape (see [0083] discloses a set of reference properties for an object is an object reference data set as stored in one or more object reference data sets 318.); and
associating, in the virtual object or in metadata of the virtual object, the properties with the virtual object (see [0099] and 0100]).
Claim 3.
The computer-implemented method of claim 2, wherein associating, in the virtual object, the area of the image with the face of the plurality of faces of the 3D shape comprises: presenting, via the user interface, the image imposed on the face, wherein boundaries of the portion of the real-world object in the image do not correspond with boundaries of the face; providing, via the user interface, an editing interface object selectable to enable editing of the image; and editing the image, responsive to receiving inputs to the editing object, so that the boundaries of the portion of the real-world object in the image correspond to the boundaries of the face. (see [0093] discloses in one example implementation of updating the 3D display, the virtual data engine 195 renders the previously created three dimensional model of the display field of view including depth data for both virtual and real objects in a Z-buffer. The real object boundaries in the Z-buffer act as references for where the virtual objects are to be three dimensionally positioned in the display as the image generation unit 120 displays just the virtual objects as the display device is an optical see-through display device. For a virtual object, the virtual data engine 195 has a target space position of where to insert the virtual object. In some examples, the virtual object target position is registered to a position of a real world object, and in other examples, the virtual object is independent of a particular real object. Also see 0092-0093], [0099] and [0129]).
Claim 4.
The computer-implemented method of claim 2, wherein the properties comprise one or more of dimensions or a weight. (see [0080], [0083], [0100] and [0099]).
Claim 5.
The computer-implemented method of claim 2, further comprising: presenting, via the user interface, a set of three-dimensional shapes including the three-dimensional shape and one or more other three-dimensional shapes. (see 0092-0093], [0099]).
Claim 6.
The computer-implemented method of claim 5, wherein the set of three-dimensional shapes include one or more of a cube, a rectangular prism, a sphere, a spheroid, a cylinder, a cone, or a pyramid. (See [0105] discloses a physical damage indicator may be updated based on shape changes to a container like broken glass or an indentation in a cylindrical can shape. Also see [0099]).
Claim 7.
The computer-implemented method of claim 1, further comprising generating the 3D virtual reset, wherein generating the 3D virtual reset comprises:
presenting, at a user interface, the 3D virtual object in the 3D virtual reset at a first position (see [0041], [0051] and [0092]);
receiving, via the user interface, an edit to the 3D virtual object in the 3D virtual reset (see [0119], [0092-0093] and [0129]);
updating the presentation of the 3D virtual reset by showing the edit (see [0104] also see [0119], [0092-0093] and [0129]); and
storing the 3D virtual reset by including, in the 3D virtual reset, information about the 3D virtual object and information about the edit. (see [0119], [0092-0093], [0104], [0123] and [0129]);
Claim 8.
The computer-implemented method of claim 7, wherein the information about the 3D virtual object comprises the 3D virtual object or a link to the 3D virtual object. (See [0041], [0051], [0092] and [0099]).
Claim 9.
The computer-implemented method of claim 7, wherein the edit comprises one or more of a change in position of the 3D virtual object in the 3D virtual reset from the first position to a second position or a rotation of the 3D virtual object. See [0120], [0122] and [0129].
Claim 10.
The computer-implemented method of claim 7, wherein the edit comprises a change to the 3D virtual object comprising one or more of changes to images of one or more faces of the 3D virtual object or a resizing of the 3D virtual object. (see [0070], [0079], [0092-0093], [0099] and [0129]).
Claim 11.
The computer-implemented method of claim 7, further comprising: storing the edited 3D virtual object at a storage location in a data storage unit, wherein the information about the 3D virtual object and the information about the edit include the edited 3D virtual object or a link to the storage location of the edited 3D virtual object. See [0070], [0079], [0092-0093], [0099] and [0129].
Claim 12.
The computer-implemented method of claim 7, further comprising: storing the edit at a storage location in a data storage unit, wherein the information about the 3D virtual object includes the 3D virtual object or a link to the 3D virtual object and the information about the edit includes the edit or a link to the storage location of the edit. See [0070], [0079], [0092-0093], [0099] and [0129].
Claim 13.
The computer-implemented method of claim 1, further comprising: responsive to receiving an input in the augmented reality environment, moving the virtual reset within the augmented reality environment so that a location within the augmented reality environment of the virtual reset corresponds to a physical location in a physical environment of a physical reset to be assembled. See [0041], [0051], [0070], [0079], [0092-0093], [0099] and [0129].
Claim 15.
The computer-implemented method of claim 1, wherein presenting the virtual reset comprises presenting the virtual rest in an augmented reality environment. (see [0032], [0041], [0051] and [0099].
Claims 16 and 20 are rejected with similar reasons as set forth in claim 1, above.
Claim 17 is rejected with similar reasons as set forth in claim 2, above.
Claim 18.
The system of claim 16, the operations further comprising generating the 3D virtual reset, wherein generating the 3D virtual reset comprises: presenting, at a user interface, the 3D virtual object in the 3D virtual reset at a first position (see [0041], [0051] and [0092]); receiving, via the user interface, an edit to the 3D virtual object in the 3D virtual reset (see [0092-0093], [0099], [0119] and [0129]); updating the presentation of the 3D virtual reset by showing the edit (see [0092-0093], [0099], [0104], [0123] and [0129]); and storing the 3D virtual reset by including, in the 3D virtual reset, information about the 3D virtual object and information about the edit. see [0092-0093], [0099], [0104], [0119] and [0129]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 14 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Geisner, and further in view of Williams et al., US 2012/0223943 A1, hereinafter Williams.
Claim 14.
Geisner does not disclose (see underlined areas) the computer-implemented method of claim 1, wherein presenting the virtual reset comprises presenting the virtual rest in a virtual reality environment.
However, Williams teaches the virtual reality environment at [0034] and fig. 2 step 210.
Thus, it would have been obvious to one of ordinary skill in the art to combine the teachings of Williams the virtual reality environment into teachings of Geisner, as it would allow the system to be completely virtual when generating the reset.
Claim 19.
Geisner does not disclose (see the underlined areas) the system of claim 16 the operations further comprising: responsive to receiving an input in the augmented reality environment, moving the virtual reset within the augmented reality environment so that a location within the augmented reality environment of the virtual reset corresponds to a physical location in a physical environment of a physical reset to be assembled.
However, Williams discloses at [0033] the effect of moving a product 700 from a first shelf to a second shelf may not be fully appreciated on a typical computer monitor. At [0051] discloses by combining the advantages of a 3D physical model in terms of ease of arrangement and comprehension of the overall store layout with the detail and point of view of the virtual model, a user is given the ability to make layout changes in context of the overall store and see in real time a visual view of the store with product-level detail. Also see [0058].
Conclusion

APPLICANT REBUTTAL: REMARKS/ARGUMENTS
Upon entry of the present amendment, claims 1-20 will be pending in this application. Claims 1-3, 7, 13-17, 19, and 20 have been amended. No claims have been canceled. No new claims have been added. No new matter is added. Based on the following remarks, Applicant respectfully requests reconsideration and allowance of the pending claims.
35 U.S.C. § 102
Claims 1-13, 15-18, and 20 stand rejected under 35 U.S.C. § 102 as allegedly being anticipated by Geisner et al., U.S. Pat. App. Pub. No. 2013/0083064 (hereinafter, "Geisner").
Claim 1
Without commenting on the merits of the rejection, Applicant has amended claim 1.
Amended claim 1 recites "storing a three-dimensional (3D) virtual object that corresponds to a real-world object, the 3D virtual object including a stored user-configured superimposition of an image area showing a stored portion of the real-world object on a face of a 3D shape,"
among other elements.
Geisner describes a head mounted display device. Geisner describes that "A location in which the resource is situated and a 3D space position or volume of the resource in the location is tracked. A property of the resource is also determined from the image data and tracked ... Responsive to notification criteria for the resource being satisfied, image data related to the resource is displayed on the near-eye AR display." (Geisner at Abstract). Further, referring to Geisner, the Office Action states that "with a display optical system 14 for each eye in which image data is projected into a user's eye to generate a display of the image data while a user also sees through the display optical systems 14 for an actual direct view of the real world."
(Office Action at page 2, Emphasis Added).
As describes above Geisner describes allowing the user to see real world objects through the display were image data is displayed. Further, the Office Action recognizes that is what Geisner describes. Thus, at best, Geisner describes a virtual object being visually overlayed on top of a physical object due to user perception through the head mounted display device.
Therefore, Geisner does not describe that a superimposition of an image area is stored, a portion of the real-world object is stored, let alone the stored superimposition being user-configurable or the stored portion of the real-world object on a face of a 3D shape. For at least the above reasons, Geisner does not disclose "storing a three-dimensional (3D) virtual object that corresponds to a real-world object, the 3D virtual object including a stored user-configured superimposition of an image area showing a stored portion of the real-world object on a face of a 3D shape."
No other cited reference discloses a "3D virtual object including a stored user- configured superimposition of an image area showing a stored portion of the real-world object on a face of a 3D shape." Thus, Applicant believes claim 1 is in condition for allowance. Since amended independent claims 16 and 20 recite similar elements as amended claim 1, Applicant believes independent claims 16 and 20 are also allowable for at least the same reasons. Dependent claims 2-15 and 17-19 are believed to be in condition for allowance because of their dependent on allowable base claims 1 and 16, respectively, and for the additional elements they recite.
Claim 2
Without commenting on the merits of the rejection, Applicant has amended claim 2.
Amended claim 2 recites "receiving, via the user interface of the device, a first indication of an input indicating properties for the 3D shape; [and] receiving, via the user interface of the device, a second indicating of an image showing a portion of a real-world object," among other elements.
Geisner discloses that an "image generation unit 120 can display a virtual object to
appear at a designated depth location in the display field of view to provide a realistic, in-focus three dimensional display of a virtual object which can interact with one or more real objects. " (Geisner at [0051]). Thus, the image displayed in Geisner is displayed based on field of view and object in the field of view. As such, the properties of the displayed image in Geisner is based on view perspective, distance, etc. Geisner does not disclose "receiving, via the user interface of the device, a first indication of an input indicating properties for the 3D shape;
[and] receiving, via the user interface of the device, a second indicating of an image showing a portion of a real-world object."
Claim 15
Amended claim 15 recites "[tihe computer-implemented method of claim 2, wherein the stored user-configured superimposition of an image area is obtained based on user interactions with one or more user interface objects that cause an image corresponding to the image area to be manipulated."
As described above, Geisner describes displaying an virtual image that overlays a physical object causing the perception of a an image on a physical object. Geisner does not discuss the superimposition of an image area being stored and user configured or that the superimposition of an image area show[s] a stored portion of the real-world object on a face of a 3D shape. For at least those reasons, Geisner necessarily does not disclose how the stored user-configured superimposition is obtained.
Further, the Office Action cites Fukushima (US 20130147839). Fukushima discloses "product information of a product recommended to be displayed in an empty space of a shelf [can be determined] based on a shelf DB storing various information related to shelves, the above-described product DB, and the like, and sends determined product information and the like to the AR providing apparatus 100." (Fukushima at [0041]). Fukushima further discloses "information of storage location of an image which three-dimensionally represents a product is stored. The image obtained by the information of storage location includes color images of the front face, side face, upper face, lower face of a product (product image information)." (Fukushima at [0063]).
At best, Fukushima discusses superimposing an image on a face of a product. However, Fukushima does not disclose a "user-configured superimposition of an image area," let alone that "the stored user-configured superimposition of an image area is obtained based on user interactions with one or more user interface objects that cause an image corresponding to the image area to be manipulated."
Thus, in addition for the independently sufficient reasons described with respect to claim 1, claim 15 is in condition for allowance because Geisner, nor Fukushima disclose "user- configured superimposition of an image area," let alone that the "stored user-configured superimposition of an image area is obtained based on user interactions with one or more user interface objects that cause an image corresponding to the image area to be manipulated."
Claim 17
Amended claim 17 recites "associating, in the 3D virtual object, the area of the image with a user-specified face of the plurality offaces of the 3D shape," among other elements.
In addition to the independently sufficient reasons described with respect to claim 1,
claim 17 is in condition for allowance because Geisner, nor any other cited reference describes "associating, in the 3D virtual object, the area of the image with a user-specified face of the plurality offaces of the 3D shape," as recited in amended claim 17.
Claim 19
Amended claim 19 recites "presenting a user interface object for performing
manipulations to the image area showing the stored portion of the real-world object, wherein the stored user-configured superimposition of the image area on the face of the 3D shape is generated based on the manipulations including at least one of: resizing, rotating, or cropping," among other elements.
In addition to the independently sufficient reasons described with respect to claim 1,
claim 19 is in condition for allowance because Geisner, nor any other cited reference describes "presenting a user interface object for performing manipulations to the image area showing the stored portion of the real-world object, wherein the stored user-configured superimposition of the image area on the face of the 3D shape is generated based on the manipulations including at least one of: resizing, rotating, or cropping," as recited in amended claim 19.
Claims 14 and 19 stand rejected under 35 U.S.C. § 103 as allegedly being unpatentable over Geisner, and further in view of Williams et al., U.S. Pat. App. Pub. No. 2012/0223943 (hereinafter, "Williams").
For the reasons described above with respect to independent claims 1 and 16, claims 14 and 19 are in condition for allowance.
CONCLUSION

NEXT OFFICE ACTION: Notice of Pre-AIA  or AIA  Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
See 112(a) rejection below:
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA  35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA  35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
In claims 1, 16 and 20 the amended sections “… stored user-configured superimposition … stored portion … ” as the subject matter not properly described in the application as filed.
What is the stored user-configured?
How does a user generate the stored user-configured superimposition (interpreted as a virtual model)?
What are the factors of an object involved in the stored user-configured?
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-13, 15-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Geisner et al., US 2013/0083064 A1, hereinafter Geisner, and further in view of  Rolion et al., US 2021/0248823 A1, hereinafter Rolion.
Claim 1.
A computer-implemented method in which one or more processing devices perform operations comprising:
storing a three-dimensional (3D) virtual object that corresponds to a real-world object, the 3D virtual object including a stored user-configured superimposition of an image area showing a stored portion of the real-world object on a face of a 3D shape (see [0035] discloses personal A/V apparatus 8 includes an optical see-through, augmented reality display device as a near-eye, augmented reality display device 2 in communication with a companion processing module 4 via a wire 6 in this example or wirelessly in other examples. In this embodiment, head mounted display device 2 is in the shape of eyeglasses in a frame 115, with a display optical system 14 for each eye in which image data is projected into a user's eye to generate a display of the image data while a user also sees through the display optical systems 14 for an actual direct view of the real world. See [0006-0007] for storing the 3D object, also see [0051]);
storing a virtual reset that includes information about the 3D virtual object and a position of the 3D virtual object in the virtual reset (see [0092] discloses the virtual data engine 195 performs translation, rotation, and scaling operations for display of the virtual data at the correct size and perspective. A virtual data position may be dependent upon, a position of a corresponding object, real or virtual, to which it is registered. The virtual data engine 195 can update the scene mapping engine about the positions of the virtual objects processed.); and
presenting the virtual reset, the presentation showing the 3D virtual object at the position (see [0041], [0051], [0127] and [0092]).
Geisner does not explicitly specify a stored user-configured superimposition of an image area.
However, Rolion teaches at [0012] placement of a three-dimensional virtual object in the registered virtual world, such that the virtual object is substantially overlaid, in the virtual world, on the real object in the real world. Also at [0032] discloses, a user can select a virtual object that has the shape closest to that of the real object. The three-dimensional virtual objects from the library of three-dimensional virtual objects may be meshes that are well-known to a person skilled in the art, or three-dimensional shapes associated with parameters. By way of illustration, a cylindrical three-dimensional shape having a circular base can be associated with a diameter parameter and a height parameter.
Thus, it would have been obvious to one of ordinary skill in the art to combine the teachings of Rolion into Geisner in order to provide the 3D virtual object including the stored user-configured superimposition of an image area showing a stored portion of the real-world object on a face of a 3D shape, see [0103] discloses an object can be selected so as to be of the same shape as the real object, or at least a shape approximate thereto.
Claim 2.
The computer-implemented method of claim 1, further comprising:
generating the 3D virtual object, wherein generating the 3D virtual object comprises: receiving, via a user interface of a device, a selection of the 3D shape that includes a plurality of faces including the face (see [0070] discloses 0070] The scene mapping engine 306 is first described. A 3D mapping of the display field of view of the augmented reality display can be determined by the scene mapping engine 306 based on captured image data, and captured depth data if available, for the display field of view. The 3D mapping includes 3D space positions or position volumes for objects. A 3D space is a volume of space occupied by the object. Depending on the precision desired, the 3D space can match the 3D shape of the object or be a less precise volume like a bounding shape like a bounding box or bounding ellipse around an object. A 3D space position represents position coordinates for the volume or 3D space. In other words the 3D space position identifies how much space an object occupies and where in the display field of view that occupied space is.);
receiving, via the user interface of the device, a first indication of an input indicating properties for the 3D shape (see [0100] FIG. 3C illustrates some examples of data fields in an object physical properties data set 320N stored for a real object, which includes data values detected or otherwise determined based on captured data of the real object, or for a specific virtual object. Also see [0070], [0079] and [0099]);
receiving, via the user interface of the device, a second indication of an image showing a portion of a real-world object (see [0041], [0051] and [0092]);
determining an area of the image that corresponds to the portion of the real-world object (see [0079], [0092] and [0099]);
associating, in the 3D virtual object, the area of the image with the face of the plurality of faces of the 3D shape (see [0083] discloses a set of reference properties for an object is an object reference data set as stored in one or more object reference data sets 318.) so the area of the image is superimposed on the face of the 3D virtual object; and
associating, in the 3D virtual object or in metadata of the virtual object, the properties with the 3D virtual object (see [0099] and 0100]).
Claim 3.
The computer-implemented method of claim 2, wherein associating, in the 3D virtual object, the area of the image with the face of the plurality of faces of the 3D shape comprises: presenting, via the user interface, the image imposed on the face, wherein boundaries of the portion of the real-world object in the image do not correspond with boundaries of the face; providing, via the user interface, an editing interface object selectable to enable editing of the image; and editing the image, responsive to receiving inputs to the editing interface object, so that the boundaries of the portion of the real-world object in the image correspond to the boundaries of the face. (see [0093] discloses in one example implementation of updating the 3D display, the virtual data engine 195 renders the previously created three dimensional model of the display field of view including depth data for both virtual and real objects in a Z-buffer. The real object boundaries in the Z-buffer act as references for where the virtual objects are to be three dimensionally positioned in the display as the image generation unit 120 displays just the virtual objects as the display device is an optical see-through display device. For a virtual object, the virtual data engine 195 has a target space position of where to insert the virtual object. In some examples, the virtual object target position is registered to a position of a real world object, and in other examples, the virtual object is independent of a particular real object. Also see 0092-0093], [0099] and [0129]).
Claim 4.
The computer-implemented method of claim 2, wherein the properties comprise one or more of dimensions or a weight. (see [0080], [0083], [0100] and [0099]).
Claim 5.
The computer-implemented method of claim 2, further comprising: presenting, via the user interface, a set of three-dimensional shapes including the three-dimensional shape and one or more other three-dimensional shapes. (see 0092-0093], [0099]).
Claim 6.
The computer-implemented method of claim 5, wherein the set of three-dimensional shapes include one or more of a cube, a rectangular prism, a sphere, a spheroid, a cylinder, a cone, or a pyramid. (See [0105] discloses a physical damage indicator may be updated based on shape changes to a container like broken glass or an indentation in a cylindrical can shape. Also see [0099]).
Claim 7.
The computer-implemented method of claim 1, further comprising generating the 3D virtual reset, wherein generating the 3D virtual reset comprises:
presenting, at a user interface, the 3D virtual object in the 3D virtual reset at a first position (see [0041], [0051] and [0092]);
receiving, via the user interface, an indication of a user selection to edit the 3D virtual object in the 3D virtual reset (see [0119], [0092-0093] and [0129]) and also see Rolion at [0032] discloses a user can select a virtual object that has the shape closest to that of the real object;
updating the presentation of the 3D virtual reset by showing the edit (see [0104] also see [0119], [0092-0093] and [0129]); and
storing the 3D virtual reset by including, in the 3D virtual reset, information about the 3D virtual object and information about the edit. (see [0119], [0092-0093], [0104], [0123] and [0129]);
Claim 8.
The computer-implemented method of claim 7, wherein the information about the 3D virtual object comprises the 3D virtual object or a link to the 3D virtual object. (See [0041], [0051], [0092] and [0099]).
Claim 9.
The computer-implemented method of claim 7, wherein the edit comprises one or more of a change in position of the 3D virtual object in the 3D virtual reset from the first position to a second position or a rotation of the 3D virtual object. See [0120], [0122] and [0129].
Claim 10.
The computer-implemented method of claim 7, wherein the edit comprises a change to the 3D virtual object comprising one or more of changes to images of one or more faces of the 3D virtual object or a resizing of the 3D virtual object. (see [0070], [0079], [0092-0093], [0099] and [0129]).
Claim 11.
The computer-implemented method of claim 7, further comprising: storing the edited 3D virtual object at a storage location in a data storage unit, wherein the information about the 3D virtual object and the information about the edit include the edited 3D virtual object or a link to the storage location of the edited 3D virtual object. See [0070], [0079], [0092-0093], [0099] and [0129].
Claim 12.
The computer-implemented method of claim 7, further comprising: storing the edit at a storage location in a data storage unit, wherein the information about the 3D virtual object includes the 3D virtual object or a link to the 3D virtual object and the information about the edit includes the edit or a link to the storage location of the edit. See [0070], [0079], [0092-0093], [0099] and [0129].
Claim 13.
The computer-implemented method of claim 1, further comprising: responsive to receiving an input in an augmented reality environment, moving the virtual reset within the augmented reality environment so that a location within the augmented reality environment of the virtual reset corresponds to a physical location in a physical environment of a physical reset to be assembled. See [0041], [0051], [0070], [0079], [0092-0093], [0099] and [0129].
Claim 15.
The computer-implemented method of claim 2, wherein the stored user-configured superimposition of the image area is obtained based on user interactions with one or more user interface objects that cause the image corresponding to the image area to be manipulated. (see [0032], [0041], [0051] and [0099].
Claims 16 and 20 are rejected with similar reasons as set forth in claim 1, above.
Claim 17 is rejected with similar reasons as set forth in claim 2, above.
Claim 18.
The system of claim 16, the operations further comprising generating the 3D virtual reset, wherein generating the 3D virtual reset comprises: presenting, at a user interface, the 3D virtual object in the 3D virtual reset at a first position (see [0041], [0051] and [0092]); receiving, via the user interface, an edit to the 3D virtual object in the 3D virtual reset (see [0092-0093], [0099], [0119] and [0129]); updating the presentation of the 3D virtual reset by showing the edit (see [0092-0093], [0099], [0104], [0123] and [0129]); and storing the 3D virtual reset by including, in the 3D virtual reset, information about the 3D virtual object and information about the edit. see [0092-0093], [0099], [0104], [0119] and [0129]).
Claims 14 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Geisner, Rolion and further in view of Williams et al., US 2012/0223943 A1, hereinafter Williams.
Claim 14.
Geisner does not explicitly disclose the computer-implemented method of claim 1, wherein presenting the virtual reset comprises presenting the virtual reset in a virtual reality environment or an augmented reality environment.
Rolion illustrates in fig. 5 a marker and information but may not present the virtual reset of the claimed invention.
However, Williams teaches the virtual reality environment at [0034] and fig. 2 step 210.
Thus, it would have been obvious to one of ordinary skill in the art to combine the teachings of Williams the virtual reality environment into teachings of Geisner and Rolion, as it would allow the system to be completely virtual when generating the reset.
Claim 19.
Geisner does not disclose (see the underlined areas) the system of claim 16 the operations further comprising: presenting a user interface object for performing manipulations to the image area showing the stored portion of the real-world object, wherein the stored user-configured superimposition of the image area on the face of the 3D shape is generated based on the manipulations including at least one of: resizing, rotating, or cropping.
However, Williams discloses at [0033] the effect of moving a product 700 from a first shelf to a second shelf may not be fully appreciated on a typical computer monitor. At [0051] discloses by combining the advantages of a 3D physical model in terms of ease of arrangement and comprehension of the overall store layout with the detail and point of view of the virtual model, a user is given the ability to make layout changes in context of the overall store and see in real time a visual view of the store with product-level detail. Also see [0058].
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action.  Accordingly, THIS ACTION IS MADE FINAL.  See MPEP § 706.07(a).  Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action.  In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action.  In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.

CITED REFERENCE SUMMARIES: Ref US20130147839A1: Claims:
An augmented reality providing system comprising an information processing terminal having an imaging unit and a display unit and an information processing apparatus which are communicable with each other, wherein the information processing terminal comprises: a request unit sending to the information processing apparatus a request, including image information of contents of imaging in the imaging unit, for obtaining product information indicating a product which can be displayed on a shelf which can be determined from the image information; and a display control unit displaying product information, included in a reply sent from the information processing apparatus in response to the request, on the display unit in a manner of overlaying on image information imaged by the imaging unit, and the information processing apparatus comprises: a determination unit determining a shelf from the image information included in the request and determining a size of an empty space of the shelf; a selection unit selecting product information of products smaller than the size of the empty space of the shelf determined by the determination unit, from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other; and a reply unit sending to the information processing terminal a reply including product information selected in the selection unit.
the information processing terminal comprises:
a request unit sending to the information processing apparatus a request, including image information of contents of imaging in the imaging unit, for obtaining product information indicating a product which can be displayed on a shelf which can be determined from the image information; and
a display control unit displaying product information, included in a reply sent from the information processing apparatus in response to the request, on the display unit in a manner of overlaying on image information imaged by the imaging unit, and
the information processing apparatus comprises:
a determination unit determining a shelf from the image information included in the request and determining a size of an empty space of the shelf;
a selection unit selecting product information of products smaller than the size of the empty space of the shelf determined by the determination unit, from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other; and
a reply unit sending to the information processing terminal a reply including product information selected in the selection unit.
The augmented reality providing system according to claim 1 , wherein the storage unit stores display rule information identifying a product allowed to be displayed on the shelf, and the selection unit selects product information of a product identified as allowed to be displayed on the shelf from the display rule information out of the product information of products smaller than the size of the empty space of the shelf.
the storage unit stores display rule information identifying a product allowed to be displayed on the shelf, and
the selection unit selects product information of a product identified as allowed to be displayed on the shelf from the display rule information out of the product information of products smaller than the size of the empty space of the shelf.
The augmented reality providing system according to claim 2 , wherein the display rule information includes category information of the product allowed to be displayed on the shelf, and the selection unit selects product information of a product belonging to the category information out of the product information of products smaller than the size of the empty space of the shelf.
the display rule information includes category information of the product allowed to be displayed on the shelf, and
the selection unit selects product information of a product belonging to the category information out of the product information of products smaller than the size of the empty space of the shelf.
The augmented reality providing system according to claim 2 , wherein the display rule information includes brand information of the product allowed to be displayed on the shelf, and the selection unit selects product information of a product of a brand corresponding to the brand information out of the product information of products smaller than the size of the empty space of the shelf.
the display rule information includes brand information of the product allowed to be displayed on the shelf, and
the selection unit selects product information of a product of a brand corresponding to the brand information out of the product information of products smaller than the size of the empty space of the shelf.
The augmented reality providing system according to claim 2 , wherein the display rule information includes product information of the product allowed to be displayed on the shelf, and the selection unit selects product information which matches the product information included in the display rule information out of the product information of products smaller than the size of the empty space of the shelf.
the display rule information includes product information of the product allowed to be displayed on the shelf, and
the selection unit selects product information which matches the product information included in the display rule information out of the product information of products smaller than the size of the empty space of the shelf.
The augmented reality providing system according to claim 1 , wherein the storage unit stores information indicating unit sales records of every product, and the selection unit preferentially selects product information of a product of which the unit sales record is relatively high out of the product information of products smaller than the size of the empty space of the shelf.
the storage unit stores information indicating unit sales records of every product, and
the selection unit preferentially selects product information of a product of which the unit sales record is relatively high out of the product information of products smaller than the size of the empty space of the shelf.
The augmented reality providing system according to claim 1 , wherein the storage unit stores information indicating unit sales records of every product, and the product information stored in the storage unit include product image information representing products, and the determination unit identifies products displayed on the shelf from the image information included in the request and the product image information stored in the storage unit, determines a product of which the unit sales record is relatively low from the products displayed on the shelf, and determines a size of a space which is generated when the determined product is removed from the shelf as the size of the empty space of the shelf.
the product information stored in the storage unit include product image information representing products, and
the determination unit identifies products displayed on the shelf from the image information included in the request and the product image information stored in the storage unit, determines a product of which the unit sales record is relatively low from the products displayed on the shelf, and determines a size of a space which is generated when the determined product is removed from the shelf as the size of the empty space of the shelf.
The augmented reality providing system according to claim 1 , wherein when it is determined that there are a plurality of empty spaces on a same level of the shelf from the image information included in the request, the determination unit determines a size obtained by adding up the plurality of empty spaces as the size of the empty space of the shelf.
when it is determined that there are a plurality of empty spaces on a same level of the shelf from the image information included in the request, the determination unit determines a size obtained by adding up the plurality of empty spaces as the size of the empty space of the shelf.
The augmented reality providing system according to claim 1 , wherein the storage unit stores end information indicating when a sale ends for every product, the information processing apparatus further comprises a creation unit creating, when it is determined based on the end information that there is product information of a product of which sale ends when a preset time passes in the product information selected in the selection unit, instruction information indicating an instruction related to the end of sale of the product, the reply unit sends the instruction information to the information processing terminal when the instruction information is created in the creation unit, and the display control unit displays the instruction information on the display unit.
the storage unit stores end information indicating when a sale ends for every product,
the information processing apparatus further comprises a creation unit creating, when it is determined based on the end information that there is product information of a product of which sale ends when a preset time passes in the product information selected in the selection unit, instruction information indicating an instruction related to the end of sale of the product,
the reply unit sends the instruction information to the information processing terminal when the instruction information is created in the creation unit, and
the display control unit displays the instruction information on the display unit.
The augmented reality providing system according to claim 1 , wherein the storage unit stores display rule information identifying a product allowed to be displayed on the shelf and product image information representing the product, the information processing apparatus further comprises a creation unit creating, when it is identified that there is a product which is not allowed to be displayed on the shelf from the display rule information and from the image information included in the request and the product image information stored in the storage unit, indication information indicating an existence of the product which is not allowed to be displayed on the shelf, the reply unit sends the indication information to the information processing terminal when the indication information is created in the creation unit, and the display control unit displays the indication information on the display unit.
the storage unit stores display rule information identifying a product allowed to be displayed on the shelf and product image information representing the product,
the information processing apparatus further comprises a creation unit creating, when it is identified that there is a product which is not allowed to be displayed on the shelf from the display rule information and from the image information included in the request and the product image information stored in the storage unit, indication information indicating an existence of the product which is not allowed to be displayed on the shelf,
the reply unit sends the indication information to the information processing terminal when the indication information is created in the creation unit, and
the display control unit displays the indication information on the display unit.
An information processing apparatus, comprising: a determination unit determining a shelf from image information of contents of imaging in an imaging unit sent from an information processing terminal having the imaging unit and a display unit and determining a size of an empty space of the shelf; a selection unit selecting product information of products smaller than the size of the empty space of the shelf determined by the determination unit, from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other; and a reply unit sending to the information processing terminal a reply including product information selected in the selection unit.
a determination unit determining a shelf from image information of contents of imaging in an imaging unit sent from an information processing terminal having the imaging unit and a display unit and determining a size of an empty space of the shelf;
An information processing terminal having an imaging unit and a display unit, the terminal comprising: a determination unit determining a shelf from image information of contents of imaging in the imaging unit and determining a size of an empty space of the shelf; a request unit sending a request, including information of the size of the empty space of the shelf determined in the determination unit, for obtaining product information indicating a product which can be displayed on the shelf to an information processing apparatus selecting product information of products smaller than the size of the empty space of the shelf from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other; and a display control unit displaying product information, included in a reply sent from the information processing apparatus in response to the request, on the display unit in a manner of overlaying on image information imaged by the imaging unit.
a determination unit determining a shelf from image information of contents of imaging in the imaging unit and determining a size of an empty space of the shelf;
a request unit sending a request, including information of the size of the empty space of the shelf determined in the determination unit, for obtaining product information indicating a product which can be displayed on the shelf to an information processing apparatus selecting product information of products smaller than the size of the empty space of the shelf from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other; and
a display control unit displaying product information, included in a reply sent from the information processing apparatus in response to the request, on the display unit in a manner of overlaying on image information imaged by the imaging unit.
An augmented reality providing method in an augmented reality providing system including an information processing terminal having an imaging unit and a display unit and an information processing apparatus which are communicable with each other, the method comprising: sending by the information processing terminal a request including image information of contents of imaging in the imaging unit to the information processing apparatus; determining by the information processing apparatus a shelf from the image information included in the request and determining a size of an empty space of the shelf; selecting by the information processing apparatus product information of products smaller than the size of the empty space of the shelf determined in the determining step, from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other; sending by the information processing apparatus a reply including product information selected in the selecting step to the information processing terminal; and displaying by the information processing terminal product information, included in a reply sent from the information processing apparatus in response to the request, on the display unit in a manner of overlaying on image information imaged by the imaging unit.
sending by the information processing terminal a request including image information of contents of imaging in the imaging unit to the information processing apparatus;
determining by the information processing apparatus a shelf from the image information included in the request and determining a size of an empty space of the shelf;
selecting by the information processing apparatus product information of products smaller than the size of the empty space of the shelf determined in the determining step, from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other;
sending by the information processing apparatus a reply including product information selected in the selecting step to the information processing terminal; and
displaying by the information processing terminal product information, included in a reply sent from the information processing apparatus in response to the request, on the display unit in a manner of overlaying on image information imaged by the imaging unit.
An information processing method, comprising: determining a shelf from image information of contents of imaging in an imaging unit sent from an information processing terminal having the imaging unit and a display unit and determining a size of an empty space of the shelf; selecting product information of products smaller than the size of the empty space of the shelf determined in the determining step, from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other; and sending to the information processing terminal a reply including product information selected in the selecting step.
determining a shelf from image information of contents of imaging in an imaging unit sent from an information processing terminal having the imaging unit and a display unit and determining a size of an empty space of the shelf;
selecting product information of products smaller than the size of the empty space of the shelf determined in the determining step, from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other; and
sending to the information processing terminal a reply including product information selected in the selecting step.
An information processing method executed by an information processing terminal having an imaging unit and a display unit, the method comprising: determining a shelf from image information of contents of imaging in the imaging unit and determining a size of an empty space of the shelf; sending a request, including information of the size of the empty space of the shelf determined in the determining step, for obtaining product information indicating a product which can be displayed on the shelf to an information processing apparatus selecting product information of products smaller than the size of the empty space of the shelf from a storage unit storing a plurality of sets of product information indicating a product and size information indicating a size of the product in association with each other; and displaying product information, included in a reply sent from the information processing apparatus in response to the request, on the display unit in a manner of overlaying on image information imaged by the imaging unit.
determining a shelf from image information of contents of imaging in the imaging unit and determining a size of an empty space of the shelf;
sending a request, including information of the size of the empty space of the shelf determined in the determining step, for obtaining product information indicating a product which can be displayed on the shelf to an information processing apparatus selecting product information of products smaller than the size of the empty space of the shelf from a storage unit storing a plurality of sets of product information indicating a product and size informat
[Truncated to input limit]