Patent Description:
<CIT> discloses superposing a dialog on an operation screen indicating a selected lighting apparatus and the peripheral area thereof. <CIT> discloses an augmented reality device for visualizing luminaire fixtures. A common lighting design method involves examining a target area with respect to floor plan, ceiling height, structures, etc. and estimating lighting for the target area using modeling tools. The modeling tools generally rely on <NUM>-D models of the target area that are created based on the examination of the target area. The generation of the <NUM>-D models of the target area and the modeling tools that use the <NUM>-D models can be quite complex. The reliability of the estimated lighting of the target area is also heavily dependent on the accuracy of the <NUM>-D models. Similar challenges also exist in IoT design. Thus, a solution that provides a user friendly and reliable means of lighting design is desirable. A similar solution can also be applied in IoT design.

The present disclosure relates generally to lighting and controls solutions, and more particularly to lighting or IoT design using augmented reality. In an example embodiment, an augmented reality-based lighting design method includes displaying, by an augmented reality device, a real-time image of a target physical area on a display screen. The method further includes displaying, by the augmented reality device, a lighting fixture <NUM>-D model on the display screen in response to a user input, where the lighting fixture <NUM>-D model is overlaid on the real-time image of the target physical area. The method also includes displaying, by the augmented reality device, a lighting pattern on the display screen overlaid on the real-time image of the target physical area, wherein the lighting pattern is generated based on at least photometric data associated with the lighting fixture <NUM>-D model.

According to a first aspect, the invention is defined by the method of claim <NUM>.

According to a second aspect, the invention is defined by the device of claim <NUM>.

These and other aspects, objects, features, and embodiments will be apparent from the following description and the appended claims.

The drawings illustrate only example embodiments and are therefore not to be considered limiting in scope. The elements and features shown in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the example embodiments. Additionally, certain dimensions or placements may be exaggerated to help visually convey such principles. In the drawings, the same reference numerals used in different drawings may designate like or corresponding, but not necessarily identical elements.

In the following paragraphs, example embodiments will be described in further detail with reference to the figures. In the description, well-known components, methods, and/or processing techniques are omitted or briefly described. Furthermore, reference to various feature(s) of the embodiments is not to suggest that all embodiments must include the referenced feature(s).

In some example embodiments, an augmented reality (AR) platform may be used by a user, such as lighting designers, consumers, builders, installers, contractors, homeowners, tenants, landlords, building occupants, etc. to place virtual fixture models into a real environment to quickly gauge appearance as well as view, coordinate, or layout various fixtures lighting parameters such as fixture aesthetic or accessory options, color temperature, shape, distribution, brightness, light levels, light beam coverage of a space or field of view (e.g., for a camera that may be integrated into the fixture) or sensor range/direction for sensors (e.g., IR or other type of motion or environmental sensor) or accessory devices (speaker range/direction, microphone range/direction) accompanying or separate from a luminaire, etc..

An AR device may include a lighting design AR application and a database of lighting fixtures along with associated photometric files or parameter data files with alternative gradient of lighting information. The photometric files (e.g., IES files) contain necessary information to estimate one or more lighting pattern(s) that is produced by lighting fixtures within a three dimensional space. The photometric files may also include color temperature, luminance intensity, and/or other information about the light emitted by a lighting fixture. The lighting design AR application enables a user to select and place one or more lighting fixtures in a real-time image of a physical/real space being displayed, for example, on a viewport of the AR device and allows visualization of how the selected lighting fixture(s) will behave and appear in the physical/real space. The AR application enables a rendered overlay of the lighting fixture and lighting patterns as well as other light characteristics (e.g., color temperature and luminosity) and accounts for reflected lighting patterns and shadows on surfaces of objects/structures in the physical/real space detected by the AR device image processing or other communication between the AR device and detected objects, which produces reasonably realistic results without requiring installation of actual lighting fixtures. For example, the AR device may implement a standalone artificial intelligence application or artificial intelligence code that is integrated with the AR application to detect and identify objects/structures in the physical/real space.

Similarly, an AR device may include a sensor (or accessory) design AR application and a database of sensors along with associated data files (range, viewing angle, resolution or similar operation information that may be visualized through the AR device). For example, the files may contain necessary information to estimate one or more view angles and range that is associated with the sensor (e.g., motion, light, temperature, humidity, sound or other type of sensor) or accessory device (e.g., camera, microphone, speaker, emitter/detector, wireless device like Bluetooth or WiFi repeater, etc.) within a three dimensional space. The files may also include other information about the light emitted by the sensor or accessory. The AR application enables a user to select and place one or more sensors or accessories in a real-time image of a physical/real space being displayed, for example, on a viewport of the AR device and allows visualization of how the selected sensors or accessories will behave and appear in the physical/real space. The AR application enables a rendered overlay of the sensors or accessories and associated patterns or visuals as well as other characteristics. The AR device may account for reflected patterns or interference based on surfaces of objects/structures in the physical/real space detected by the AR device image processing or other communication between the AR device and detected objects, which produces reasonably realistic results without requiring installation of actual sensors or accessories.

<FIG> illustrate an augmented reality device <NUM> for lighting design according to an example embodiment. In some example embodiments, <FIG> illustrates a back side of the augmented reality device <NUM>, and <FIG> illustrates the front side of the augmented reality device <NUM>. For example, the augmented reality device <NUM> may be a tablet, a smartphone, etc. Alternatively, the augmented reality device <NUM> may be a headset, glasses, goggles, or another type of device with an augmented reality capable display.

Referring to <FIG>, in some example embodiments, the augmented reality (AR) device <NUM> may include a back-facing camera <NUM> on a back side of the augmented reality device <NUM>. The AR device <NUM> may also include a viewport/display screen <NUM> on a front side of the augmented reality device <NUM>. In some example embodiments, the AR device <NUM> includes an ambient light sensor <NUM>, and may also include a front-facing camera <NUM>, a user input area <NUM>, accelerometers, or other sensors useful in determining orientation or real-time feedback from the physical space the AR device <NUM> is located for use in interpreting and displaying the AR on the display <NUM> of the AR device <NUM>.

In some example embodiments, the viewport <NUM> may be used to display images as seen by the cameras <NUM>, <NUM> as well as to display objects (e.g., icons, text, etc.) stored, received, and/or generated by the AR device <NUM>. The viewport <NUM> may also be used as a user input interface for the AR device <NUM>. For example, the viewport <NUM> may be a touch sensitive display screen. The viewport <NUM> may contain a number of pixels in the vertical and horizontal directions (known as display resolution). For example, the viewport <NUM> may have a display resolution of <NUM> x <NUM>. Each pixel may contain subpixels, where each subpixel typically represents red, green, and blue colors.

In some example embodiments, an image of a physical/real area in front of the AR device <NUM> may be displayed on the viewport <NUM> in real time as viewed by the camera <NUM>. For example, the AR device <NUM> may include a lighting design AR application that activates the camera <NUM> such that a real-time image of the physical space viewed by the camera <NUM> is displayed on the viewport <NUM>. Alternatively, the camera <NUM> may be enabled/activated to display a real-time image of the physical space before or after the lighting design AR application started. In some example embodiments, the real-time image displayed on the physical space may be displayed with a slight delay.

In some example embodiments, the AR device <NUM> may include an artificial intelligence application and/or component that can determine real light emitting surfaces and/or other surfaces or structures, such as windows, ceilings, walls, floors, mirrored or reflective surfaces, etc. in a physical space/area, and automatically suggest/provide recommended types of lighting fixtures along with additional information such as suggested location, orientation, and/or an appropriate number of lighting fixtures based on characteristics associated with the light fixtures (e.g., glare, intensity, available color temperatures or colors, available optics or accessories that change the beam angle or distribution produced by the light fixture, etc.). For example, the artificial intelligence software application and/or component may identify or suggest the right location for a certain fixture in the observed space, which results in requiring minimal input, interaction, and decision making by a user in achieving lighting design of a physical space/area. Similarly, a software application incorporating suggestions or that identifies suggested locations for devices such as sensors (motion, light, environmental conditions like heat, humidity, sound, etc.) or accessories (e.g., cameras, microphones, speakers, wireless communication, repeaters, etc.) could be used in embodiments aimed at sensors or accessories instead of or in addition to light fixtures.

<FIG> illustrate augmented reality devices <NUM>, <NUM> for lighting design and IoT design according to another example embodiment. In some example embodiments, the AR device <NUM> may be used to perform the operations described above with respect to the AR device <NUM> in a similar manner. For example, the glass screens of the devices <NUM>, <NUM> may be used as display screens similar to the viewport <NUM> of the AR device <NUM>. In some example embodiments, another AR device may be used to perform the operations performed by the AR device <NUM> in a similar manner as described above with respect to <FIG>. Although the descriptions below are presented generally with respect to the AR device <NUM> of <FIG>, the description is equally applicable to the AR devices <NUM>, <NUM> of <FIG>.

<FIG> illustrates a block diagram of the augmented reality device <NUM> of <FIG> according to an example embodiment. In some example embodiments, the block diagram of <FIG> may correspond to the augmented reality devices <NUM>, <NUM> of <FIG>. Referring to <FIG>, and <FIG>, in some example embodiments, the AR device <NUM> includes a controller <NUM>, a camera component <NUM>, a display component <NUM>, an input interface <NUM>, a memory device <NUM>, and a communication interface <NUM>. For example, the camera component <NUM> may correspond to or may be part of the cameras <NUM>, <NUM>. The display component <NUM> may correspond to or may be part of the viewport/display screen <NUM> and may include circuitry that enables or performs displaying of information (e.g., images, text, etc.) on the viewport <NUM>. For example, the pixels of the viewport may be set/adjusted to display the image as viewed by the camera <NUM> or <NUM>. The input interface <NUM> may correspond to the user input area <NUM> and/or the user input capability of viewport <NUM>. For example, the display component <NUM> and the input interface <NUM> may make up or may be part of the viewport <NUM>, where the viewport <NUM> is, for example, a touch-sensitive display screen. The communication interface <NUM> may be used for communication, wirelessly or via a wired connection, by the AR device <NUM>.

The controller <NUM> may include one or more microprocessors and/or microcontrollers that can execute software code stored in the memory device <NUM>. For example, the software code of the lighting design AR application and IoT design application may be stored in the memory device <NUM> or retrievable from a remote storage location (e.g., cloud service or remotely located server or database) via the communication interface <NUM> or other communication means. Other executable software codes used in the operation of the AR device <NUM> may also be stored in the memory device <NUM> or in another memory device of the AR device <NUM>. For example, artificial intelligence lighting and/or other software may be stored in the memory device <NUM> as part of the AR application or along with the AR application and may be executed by the controller <NUM>.

To illustrate, the controller <NUM> may execute the artificial intelligence application to determine real light emitting surfaces and/or structures (e.g., windows), reflective surfaces, etc. in a physical space/area, for example, based on the real-time image of the physical space/area as viewed by the camera <NUM> or <NUM> and/or based on lighting condition sensed by an ambient light sensor component <NUM> (corresponding to, connected to, or included in the ambient light sensor <NUM>), and automatically suggest/provide recommended type(s) of lighting fixtures along with additional information such as suggested location, orientation, and/or an appropriate number of lighting fixtures. In general, the one or more microprocessors and/or microcontrollers of the controller <NUM> execute software code stored in the memory device <NUM> or in another device to implement the operations of the AR device <NUM> described herein. In some example embodiments, the memory device <NUM> may include a non-volatile memory device and volatile memory device.

In some example embodiments, data that is used or generated in the execution of the lighting design AR application, IoT design AR application, and other code may also be retrieved and/or stored in the memory device <NUM> or in another memory device of the AR device <NUM> or retrieved from a remote storage location (e.g., cloud service or remotely located server or database) via the communication interface <NUM> or other communication means. For example, <NUM>-D models of lighting fixtures and photometric data files (e.g., IES files) associated with the lighting fixture models may be stored in the memory device <NUM>, or retrieved from storage on a remote "cloud"-based service, and may be retrieved during execution of the lighting design AR application. <NUM>-D models of other devices such as sensors, cameras, microphones, speakers emitter/detector, wireless devices such as Bluetooth or WiFi repeater, etc. and data associated with the devices may be stored in the memory device <NUM>, or stored in and retrieved from storage on a remote "cloud"-based service, and may be retrieved during execution of IoT design AR application on the AR device <NUM>.

The data stored and/or retrieved may include information such as range, viewing angle, resolution or similar operation information that may be visualized through the AR device). For example, the data may contain necessary information to estimate one or more view angles and range that is produced by sensor (e.g., motion, light, temperature, humidity, sound or other type of sensor) or an accessory device, such as camera, microphone, speaker, emitter/detector, wireless device like Bluetooth or WiFi repeater, etc. within a three dimensional space. The files may also include other information about the light emitted by the sensor or the accessory device.

In some example embodiments, the lighting design AR application stored in the memory device <NUM> may incorporate or interface with an augmented reality application/software, such as ARKit, ARCore, HoloLens, etc., that may also be stored in the memory device <NUM> or called upon from or provided via a remote storage location (e.g., cloud service or remotely located server or database) via the communication interface <NUM> or other communication means.

The controller <NUM> may communicate with the different components of the AR device <NUM>, such as the camera component <NUM>, etc., and may execute relevant code, for example, to display a real-time image as viewed by the camera <NUM> and/or <NUM> as well as other image objects on the viewport <NUM>.

Although the block diagram of <FIG> is described above with respect to the AR device <NUM>, the block diagram and the above description are equally applicable to the AR devices <NUM>, <NUM> of <FIG>.

<FIG> illustrate lighting design stages using the augmented reality devices of <FIG> according to an example embodiment. Although the descriptions below are presented generally with respect to the AR device <NUM> of <FIG>, the description is equally applicable to the AR devices <NUM>, <NUM> of <FIG>. In some example embodiments, <FIG> illustrates a real-time image <NUM> of a target area <NUM> displayed on the AR device <NUM> incorporating the lighting design AR application. To illustrate, after the lighting design AR application is started, for example, by selecting a lighting design AR application icon displayed on the viewport <NUM>, a real-time image <NUM> of the target area <NUM> may be displayed on the viewport <NUM>. The real-time image <NUM> displayed on the viewport <NUM> may be an image of the target area <NUM> as viewed by the back-facing camera <NUM>. For example, a sofa <NUM> and a lighting fixture <NUM> that are real objects in the target area <NUM> are shown in the real-time image <NUM>. The back-facing camera <NUM> may be enabled/activated to view (not necessarily record) the target area <NUM> in response to the activation of the lighting design AR application or may be enabled/activated separately.

In some example embodiments, the AR device <NUM> may be used to assess the target area <NUM> to identify objects, structures, surfaces, etc. in the target area <NUM>. For example, the AR device <NUM> may include and use one or more accelerometers to determine the orientation of the AR device <NUM> relative to the target area <NUM>, and thus determine orientation of objects, structures, surfaces, etc. in the target area <NUM> based on the real-time image <NUM> of the target area <NUM> as captured by the camera <NUM>. The AR device <NUM> may identify objects, structures, surfaces, etc. by executing artificial intelligence and image processing code and based on lighting condition of the target area sensed by the ambient light sensor <NUM>. For example, the AR device <NUM> may identify light reflective (e.g., mirror), transmissive (e.g., windows), ceilings, walls, floor, furniture, etc. based on the real-time image <NUM> of the target area <NUM>, the lighting conditions of the target area <NUM>, the orientation of the AR device <NUM>, etc. The AR device <NUM> may use information from the assessment of the target area <NUM>, for example, to generate display models representing the lighting pattern(s) resulting from selected lighting fixture models as described below.

In some example embodiments, <FIG> illustrates a modified image <NUM> of the target area <NUM> displayed on the viewport <NUM> of the AR device <NUM>. For example, a user may provide an input to the AR device <NUM> (e.g., via the input area <NUM> or via the viewport <NUM>) to apply a darkening filter to the pixels of the viewport <NUM> such that the modified image <NUM> is a result of the real-time image <NUM> and the darkening of the viewport <NUM>. As can be seen in <FIG>, the real-time image <NUM> of the target area <NUM> may still be visible to the user after the darkening filter is applied. To illustrate, the darkening of the viewport <NUM> may provide a reference lighting level to allow subsequent adjustments of lighting pattern and other characteristics to be more easily discernable.

During the application of the darkening filter to the viewport <NUM>, the pixels of the viewport <NUM> are transformed based on the pixel data from the camera <NUM> (i.e., the real-time image viewed by the camera <NUM>) and the light level detected by the ambient light sensor <NUM>. In some example embodiments, to darken the pixels of the viewport <NUM>, the lighting design AR application may include code corresponding to the equation shown below that is executed by the AR device <NUM> with respect to the individual pixels of the viewport <NUM>: <MAT>.

By considering the ambient light level, the viewport <NUM> may be darkened to a level that allows the real-time image <NUM> of the target area <NUM> to be viewed by user. After the viewport is darkened, the lighting design AR application may display a message to the user indicating the option of displaying or adding lighting fixtures to the modified image <NUM>.

In some example embodiments, <FIG> illustrates the modified image displayed on the viewport <NUM> of the AR device <NUM> along with lighting fixture <NUM>-D models menu <NUM> for selection of one or more lighting fixture <NUM>-D models by a user. As described above, each light fixture <NUM>-D model may be stored in a database and may be associated with a photometric file (e.g., IES file) that includes information indicating lighting pattern, color temperature, luminance intensity, etc. In some example embodiments, the lighting fixture <NUM>-D models selectable through the menu <NUM> may include different models of the same type of lighting fixture and/or different types of lighting fixtures, where the different models are associated with respective photometric files representing different lighting patterns, color temperatures, luminance intensity, etc..

In general, the light fixture <NUM>-D models selectable through the menu <NUM> may be provided to the user for selection in one of several other means such as by displaying the models at other locations on the viewport <NUM>, separately on a different display page, as drop-down menu items, etc. Alternatively, the light fixture <NUM>-D models can be selected prior to bringing up the viewport <NUM> to display the selected light fixture <NUM>-D models in the viewed space.

In accordance with some example embodiments, <FIG> illustrates four lighting fixture <NUM>-D models <NUM>-<NUM> displayed on the viewport <NUM> along with the image shown in <FIG> and <FIG>. To illustrate, a user may select four lighting fixture <NUM>-D models from the lighting fixture models menu <NUM> provided to the user as shown in <FIG>. A user may select the lighting fixture <NUM>-D model <NUM> from the lighting fixture models menu <NUM> shown in <FIG> and place the model <NUM> at a desired location on the modified image. For example, a user may use a finger, stylus, or a mouse to select and place the model <NUM> at a desired location. The user may select and place the other <NUM>-D lighting fixture <NUM>-D models <NUM>-<NUM> at desired locations on the modified image in a similar manner as the model <NUM>, resulting in the image <NUM> displayed in the viewport <NUM> shown in <FIG>. A user may remove one or more of the models <NUM>-<NUM> from the viewport <NUM>, for example, by dragging the particular one more models of the viewport <NUM> or by other means as may be contemplated by those of ordinary skill in the art with the benefit of this disclosure.

In some example embodiments, when a user places the lighting fixture <NUM>-D models <NUM>-<NUM> at the locations on the modified image, the lighting fixture <NUM>-D models <NUM>-<NUM> are associated with physical locations in the target area <NUM> such that the lighting pattern resulting from the selected lighting fixture models <NUM>-<NUM> is shown relative to the physical locations in the target area <NUM>. For example, the AR device <NUM> may use display coordinates of the viewport <NUM> to keep track of the physical locations of the target area corresponding to the locations on the modified image. The AR device <NUM> may track one or more of tilt angle, orientation, direction, location, distance, etc. of the AR device <NUM> to keep the viewport <NUM> associated with physical locations of the target area <NUM>.

In some example embodiments, <FIG> illustrates an image <NUM> of the target area <NUM> overlaid with a lighting pattern resulting from the selected lighting fixture <NUM>-D models <NUM>-<NUM> and associated photometric files. The AR device <NUM> executes the lighting design AR application to process the photometric data of each selected <NUM>-D model <NUM>-<NUM> and generate the lighting pattern and the <NUM>-D models <NUM>-<NUM> overlaid on the real-time image <NUM> of the target area <NUM>. In some alternative embodiments, another device, such as a local or remote (e.g., cloud) server, may execute some of the functions of the lighting design AR application such as the processing of the photometric data, and provide the resulting information to the AR device <NUM>.

In some example embodiments, the lighting design AR application selectively removes/changes the darkening filter applied to the pixels, as necessary, based on the photometric profile (e.g., IES) of the selected lighting fixture <NUM>-D models <NUM>-<NUM>. To illustrate, the pixels of the viewport <NUM> may be selectively brightened based on the photometric data corresponding to the selected lighting fixture <NUM>-D models <NUM>-<NUM>. For example, pixels of the viewport <NUM> that are in the lighting distribution area of the selected lighting fixture <NUM>-D models <NUM>-<NUM> may be brightened in contrast to the modified image <NUM> shown in <FIG>.

In some example embodiments, the lighting pattern as determined by the AR device <NUM> may include an area <NUM> that is well lit as compared to areas <NUM> and <NUM> that may be dimly lit. For example, the areas <NUM>, <NUM> may be lit primarily as a result of reflected light from the lights produced by the selected lighting fixture <NUM>-D models <NUM>-<NUM>. To illustrate, the lighting design AR application may process the photometric data of the selected <NUM>-D model <NUM>-<NUM> to determine areas that may be lit directly and/or as a result of reflected light. The lighting design AR application may process the photometric data of the selected <NUM>-D model <NUM>-<NUM> to determine the appearance of shadows on detected or determined surfaces/objects in the real-time image <NUM> of the target area <NUM>, resulting in realistic lighting patterns. For example, the AR device <NUM> may execute an artificial intelligence application to determine objects and structures in the target area, for example, based on the real-time image of the target area as viewed by the camera of the AR device <NUM>. For example, the AR device <NUM> may identify reflective surfaces, walls, furniture, etc. and account for reflections, shadows, etc. in removing/changing the darkening filter applied to the pixels of the viewport <NUM>. In all embodiments, the AR device <NUM> also accounts for the lighting conditions in the target area, for example, based on lighting conditions sensed by the ambient light sensor <NUM>. For example, the AR device <NUM> may use the lighting condition in the target area to set/adjust parameters used in removing/changing the darkening filter applied to the pixels of the viewport <NUM>.

In some example embodiments, the AR device <NUM> may use the photometric data associated with each selected lighting fixture <NUM>-D model <NUM>-<NUM> to generate a lighting display model of the lighting pattern that is overlaid on the real-time image of the target area, resulting in the image <NUM> shown in <FIG>. The display model of the lighting pattern (including luminance levels, color temperature, etc.) may be a polygon or another type of image. In some alternative embodiments, the AR device <NUM> may send information indicating the selected lighting fixture <NUM>-D models <NUM>-<NUM> to another processing device, such as a local or cloud server, and the other processing device may generate a display model (e.g., a polygon or another image) based on the photometric data associated with the respective lighting fixture <NUM>-D models <NUM>-<NUM>. The AR device <NUM> may receive or retrieve the generated display model from the other processing device for display on the viewport <NUM>, where the display model is overlaid on the real-time image of the target area <NUM>.

In some example embodiments, the display model may be a polygon, such as a <NUM>-dimensional (2D) polygon, a <NUM>-dimensional (<NUM>-D) polygon, a combination of 2D and/or <NUM>-D polygons, etc., or one or more other types of images such as graphical images, etc. To illustrate, the image displayed in <FIG> may be a result of the AR device <NUM> overlaying a lighting display model over the image <NUM> shown in <FIG>, which effectively removes/changes the darkening filter shown in <FIG>. For example, the AR device <NUM> may generate or retrieve a polygon that has display parameters corresponding to the lighting pattern represented by the photometric data files associated with the multiple selected lighting fixture <NUM>-D models <NUM>-<NUM>. Information such as color temperature, luminance levels, etc. contained in the photometric data files may be represented by the display parameters of the polygon, and the pixels of the viewport <NUM> may be changed/set based on these parameters. Different points or parts of the generated polygon may be associated with different luminance levels, color temperature values, etc. contained in the photometric data files. The AR device <NUM> may display the real-time image of the target area overlaid with the polygon by adjusting/setting the pixels of the viewport <NUM>. For example, the display of the generated polygon on the viewport <NUM> may remove/change the darkening filter applied at the design stage shown in <FIG>, resulting in the image shown in <FIG>.

In some example embodiments, the AR device <NUM> may generate or retrieve a display model, such as a polygon (e.g., a 2D polygon, a <NUM>-D polygon, a combination of 2D and/or <NUM>-D polygons, graphical images, etc.) or another type of image(s), for each one of the selected lighting fixture <NUM>-D model <NUM>-<NUM> and combine the multiple display models to generate a display model representing the combined lighting pattern. For example, the AR device <NUM> may combine polygons that have parameters corresponding to the photometric data of each selected lighting fixture <NUM>-D model <NUM>-<NUM> to generate a combined polygon that has display parameters that account for the display parameters of the individual polygons. The AR device <NUM> may retrieve the individual polygons or other types of display models from a local storage or a remote source such as a cloud server.

In all embodiments, the AR device <NUM> accounts for lighting conditions in the target area in generating the display model representing the lighting pattern resulting from the selected lighting fixture <NUM>-D model <NUM>-<NUM>. For example, the AR device <NUM> may use the lighting condition sensed by the ambient light sensor <NUM> as well as the photometric data of each selected lighting fixture <NUM>-D model <NUM>-<NUM> to generate the display parameters of a polygon that is displayed on the viewport <NUM> overlaid on the real-time image of the target area <NUM>. The AR device <NUM> may identify reflective surfaces, walls, furniture, etc. as described above and account for reflections, shadows, etc. in generating the polygon that is overlaid on the real-time image.

As illustrated in <FIG>, the selected lighting fixture <NUM>-D models <NUM>-<NUM> are displayed in the real-time image of the target area <NUM>, enabling the user to assess how the corresponding lighting fixtures or lighting effect will look when installed in the target area <NUM>. Using the AR device <NUM>, a user (e.g., a lighting designer, owner, etc.) may more effectively perform lighting design of a particular area (e.g., a living room, a bedroom, a hallway, office, warehouse, an outdoor landscape, a parking lot, etc.) without having to install actual lighting fixtures and at the same time minimizing design errors. Because the selected lighting fixture models <NUM>-<NUM> are associated with the physical locations of the target area <NUM> as described above and because the lighting display models (e.g., the polygon(s)) are associated with the selected lighting fixture models <NUM>-<NUM>, a user may move in the target area <NUM> holding the AR device <NUM> and assess the placements of the lighting fixtures and the resulting lighting effect at different locations in the target area <NUM>. As the user moves through and near the target area <NUM>, the shape of the lighting pattern displayed on the viewport <NUM> may change depending on the part of the target area viewable by the camera <NUM> of the AR device <NUM> and the corresponding real-time image displayed on the viewport <NUM>.

As described above, a display model that represents the photometric data associated with one or more lighting fixtures may be a 2D polygon, a <NUM>-D polygon, and a combination of 2D and/or <NUM>-D polygons, graphical image(s), another type of image(s), etc. In general, a polygon that is used as a display model may be a 2D polygon, a <NUM>-D polygon, a combination of 2D and/or <NUM>-D polygons, graphical image(s), another type of image(s), etc..

In some example embodiments, a user may change the outward appearances (e.g., color) of the lighting fixture <NUM>-D models <NUM>-<NUM> without changing lighting characteristics (e.g., luminance level, color temperature, etc.) associated with the lighting fixture <NUM>-D models <NUM>-<NUM>. For example, in response to a user input (e.g., clicking or tapping on a displayed lighting fixture <NUM>-D model), the AR device <NUM> may change the color of the trim ring and/or the color of the housing of the displayed lighting fixture <NUM>-D model without changing the lighting pattern displayed on the viewport <NUM>. For example, clicking or tapping on a displayed lighting fixture <NUM>-D model by a user may result in the AR device <NUM> executing software code to change the color of the housing in a predefined order (e.g., white, blue, red, white,.

In some example embodiments, a user may use the AR device <NUM> to assess the appearance of the corresponding lighting fixtures in the target area <NUM>. For example, the AR device <NUM> may overlay the lighting fixture <NUM>-D models <NUM>-<NUM> in the real-time image <NUM> of the target area <NUM> to assess the appearance of the corresponding lighting fixtures in the target area <NUM> without installing the lighting fixtures. To illustrate, after the real-time image <NUM> is displayed on the viewport <NUM> as shown in <FIG>, the AR device <NUM> may overlay the lighting fixture <NUM>-D models <NUM>-<NUM> on the real-time image <NUM> in response to a user input. For example, the lighting fixture <NUM>-D models menu <NUM> may be displayed on the viewport <NUM> along with the image <NUM>. A user may select and place the lighting fixture <NUM>-D models <NUM>-<NUM> and/or other <NUM>-D models on the real-time image <NUM>. In some example embodiments, the lighting patterns associated with the lighting fixture <NUM>-D models <NUM>-<NUM> and other <NUM>-D models may or may not be displayed on the viewport <NUM> when the AR device <NUM> is used to assess the physical appearance of lighting fixtures. For example, the design stages associated with <FIG> and subsequent generation and/or display of a lighting pattern may be omitted.

As described above, the color of a trim ring, size of the trim ring, type of trim ring or alternative optical attachment, lens type, the color of a lighting fixture housing, alternative subcomponent(s) of the light fixture, and/or other aesthetic aspects of a displayed lighting fixture <NUM>-D model may be changed, for example, by tapping or clicking on the displayed lighting fixture <NUM>-D model. In some alternative embodiments, aesthetic features of displayed lighting fixture <NUM>-D models, such as the <NUM>-D models <NUM>-<NUM>, may be changed after the lighting patterns associated with the lighting fixture <NUM>-D models are displayed, for example, as shown in <FIG>.

In general, the lighting design AR application executed by the AR device <NUM> may include or rely on operations performed by AR applications, such as ARKit, ARCore, etc. In some alternative embodiments, a still image (a captured picture) of the target area <NUM> may be used instead of a real-time image. For example, a photograph that contains adequate information, such as tilt angle of the AR device <NUM>, GPS location, etc. may allow the AR device <NUM> executing the lighting design AR application and/or an artificial intelligence application to determine <NUM>-D information from the photograph and enable lighting design based on the information.

In some alternative embodiments, another device may perform some of the operations described herein with respect to the AR device <NUM>. To illustrate, another device, such as a local or remote server, may generate one or more display models based on information provided by the AR device <NUM>. For example, the AR device <NUM> may provide information such as the selected lighting fixture <NUM>-D model <NUM>-<NUM> and/or relevant photometric data to another processing device that generates the display model(s), and the AR device <NUM> may receive/retrieve the generated display model(s) from the other processing device.

<FIG> illustrates luminance levels indicated on the viewport <NUM> of the AR device <NUM> of <FIG> according to an example embodiment. In some example embodiments, luminance level values may be displayed on the viewport <NUM>, for example, to provide a numeric representation of brightness levels at different locations of the target area based on the selected lighting fixture <NUM>-D model <NUM>-<NUM>. For example, different points or areas of a display model (e.g., different points or areas of a polygon) generated as described above may be associated or otherwise tagged with the luminance level values. To illustrate, some areas may be associated with higher brightness level (e.g., <NUM> foot-candle (FC)) while other areas may be associated with a relatively darker level (e.g., <NUM> FC). As a user moves in the target area holding the AR device <NUM>, the luminance level values that are displayed may change depending on the part of the target area that is viewed by the camera of the AR device <NUM> and displayed on the viewport <NUM> based on the location of the user relative to the selected lighting fixture <NUM>-D model <NUM>-<NUM>.

<FIG> illustrates a <NUM>-D model of a lighting fixture <NUM> and lighting pattern including luminance levels that are based on photometric data or another gradient of lighting data associated with the lighting fixture according to an example embodiment. For example, the lighting fixture <NUM> may correspond to the lighting fixtures <NUM>-<NUM> shown in <FIG>. The photometric data associated with the lighting fixture <NUM> may be illustrated to convey lighting distribution shape, color temperature as well as the luminance levels indicated by the luminance level values, for example, at a surface that is a particular distance from the lighting fixture <NUM>. Although the luminance level values are shown for a particular surface, the photometric data may include luminance level values at different distances. The AR device <NUM> may use the photometric data including lighting distribution shape, color temperature, the luminance levels, etc. to generate a display model that is overlaid on the real-time image of the target area displayed on the viewport <NUM>. Although a polygon is described herein as an example of a display model, other types of display models such as other types of images may equally be used.

<FIG> illustrates an e-commerce interface <NUM> displayed on the augmented reality device <NUM> of <FIG> according to an example, beyond the claimed invention. Referring to <FIG>, the information included in the e-commerce interface <NUM> may be generated based on the lighting design stages described above. To illustrate, in some examples, the user may be given an option to purchase the lighting fixtures corresponding to the selected lighting fixture <NUM>-D models <NUM>-<NUM>. For example, the AR device <NUM> may execute the AR application to display a weblink on the viewport <NUM> for a user to click or tap to purchase the lighting fixtures corresponding to the selected lighting fixture <NUM>-D models <NUM>-<NUM>. Alternatively, the weblink may be provided to the user on a separate web browser page or in a separate e-commerce application screen when the design stages are completed and/or the display of AR related information is terminated by the user. Other purchasing options including the option to make the purchase via voice command, etc. may also be provided to the user. For example, the lighting design AR application may incorporate or interface with another application to provide the purchasing option as well as to execute the purchase of the lighting fixtures based on the user's input.

In some alternative examples, the e-commerce interface <NUM> may be displayed in a different format than shown in <FIG>. In some alternative examples, other user input icons and information may be displayed on the viewport without departing from the scope of this disclosure. Although the e-commerce interface <NUM> is described above with respect to the AR device <NUM> of <FIG>, the description is equally applicable to the AR devices <NUM>, <NUM> of <FIG>.

<FIG> illustrates a bill of material (BOM) generation input interface <NUM> displayed on the augmented reality device <NUM> of <FIG> according to an example, beyond the claimed invention. In some examples, a user may use the BOM generation input interface <NUM> to generate a BOM (or purchase order for the BOM) or, more generally, a list of products available for purchase (including, in some embodiments, any accessories or additional items required for installation or operation) resulting from the AR-based design described above. For example, following the display of the image <NUM> shown in <FIG>, the BOM generation page may be displayed in the viewport <NUM> as shown in <FIG>. A user may tap or click on the BOM generation input interface <NUM>, and, in response, the AR device <NUM> may execute the AR application or another application to generate a BOM that includes, for example, identification information (e.g., model number, product number, etc.) that corresponds to the lighting fixture <NUM>-D models <NUM>-<NUM> shown in <FIG> and/or other lighting fixtures and devices added by the user (including, in some embodiments, any accessories or additional items required for installation or operation). The BOM generation page shown in <FIG> may be presented on the viewport <NUM> prior to the display of the e-commerce interface <NUM> shown in <FIG>. For example, the e-commerce interface <NUM> shown in <FIG> may be displayed on the viewport <NUM> following the generation of a BOM.

In some examples, a product menu <NUM> may also be displayed on the viewport <NUM>. For example, the product menu <NUM> may allow a user to add additional products to a BOM. The product menu <NUM> may allow a user to add lighting fixtures with or without integrated IoT devices (e.g., sensors, camera, speakers, microphones, etc.), load control devices (e.g., relays, switches, dimmers, etc.), IoT devices (e.g., standalone connected sensors, microphones, a speaker, etc.), trims, junction boxes, wall-stations, and other types of products and any accessories or additional items required for installation or operation (e.g., wire harness, connectors, cables, remote power supplies, etc.) to the generated BOM. As used herein IoT device refers to any sensor and/or communication device that may be integrated into a light fixture or may be a standalone device that is capable of controlling or otherwise communicating with or to a light fixture or other device located in the vicinity of the IoT device or providing communications for a light fixture or other device in the vicinity of the IoT device to a network. Alternatively or in addition, the product menu <NUM> may allow a user to add additional products prior to the generation of a BOM. To illustrate, following the design stages corresponding to <FIG> or <FIG>, a user may add other products (e.g., a load control device, etc.) using the product menu <NUM> prior to the generation of a BOM.

In some examples, the product menu <NUM> may be a drop down menu, another type of user interface (e.g., a list), a link to another page, etc. In some examples, a product search interface may also be presented instead of or in addition to the product menu <NUM>. In some alternative embodiments, the BOM generation input interface <NUM> may be displayed on the viewport <NUM> at different design stages such as at the design stages corresponding to <FIG>. In some alternative examples, the BOM generation input interface <NUM> may be displayed at a different location of the viewport <NUM> and/or or may be displayed or provided to the user in a different format such as a selection from a drop-down menu, etc..

<FIG> illustrates a bill of material (BOM) <NUM> displayed on the augmented reality device <NUM> of <FIG> according to an example, beyond the claimed invention. Referring to <FIG>, in some examples, the BOM <NUM> may be generated by the AR device <NUM> in response to the user input provided via the BOM generation input interface <NUM>. For example, a user may use the BOM generation input interface <NUM> displayed on the viewport <NUM> as shown in <FIG>, or as can be displayed at other design stages such as the design stages corresponding to <FIG> and <FIG>.

In some examples, after the BOM <NUM> is generated and displayed, a user may add additional products such as lighting fixtures with or without integrated IoT devices, load control devices, IoT devices, trims, junction boxes, wall-stations, and other types of products to the generated BOM <NUM>. For example, a user may use the product menu <NUM> to add additional products to the generated BOM <NUM> as described above with respect to <FIG>.

In some examples, a user may request validation of the BOM <NUM> by providing an input using the BOM validation input interface <NUM>. For example, clicking or tapping the BOM validation input interface <NUM> may result in the BOM <NUM> being sent to a technical support person, a contractor, a sales representative, or automated validation system in communication with the AR device that can confirm the accuracy, completeness, or availability of the items listed on the BOM. The transmission of the BOM <NUM> by the AR device <NUM> may be performed by executing the AR application and/or another software code or application. Alternatively or in addition to sending the BOM <NUM>, clicking or tapping the BOM validation input interface <NUM> may initiate a chat session with a technical support person, a contractor, a sales representative, etc..

In some examples, clicking or tapping the BOM validation input interface <NUM> may initiate operations by the AR device <NUM> to verify design information <NUM>, which may include whether the products included in the BOM <NUM> are compliant with one or more lighting or electrical codes and/or guidelines. For example, the lighting or electrical codes and/or guidelines may be international, national, and/or local codes and guidelines. To illustrate, the lighting or electrical codes and/or guidelines may address light levels relevant to particular spaces (e.g., OSHA guidelines, etc.), lighting fixture standby power and startup time (e.g., Title <NUM> of the California Code of Regulations, etc.), plenum rating (e.g., City of Chicago Electrical Code, etc.), and other electrical and lighting requirements and guidelines such as those included in European Union standards.

In some examples, one or more lighting and/or electrical codes and/or guidelines may be stored in the memory device <NUM> or another memory device. Alternatively or in addition, one or more lighting and/or electrical codes and/or guidelines may be retrieved or compared for compliance by the AR device <NUM> from a remote source in response to a user input provided to the AR device <NUM> via the BOM validation input interface <NUM> or another user interface. For example, the AR device <NUM> may retrieve relevant lighting and/or electrical code and/or guidelines or compare compliance with such guidelines based on geographic location information provided by a user or based on a location of the AR device <NUM> determined by the AR device <NUM> using GPS and/or other means.

In some examples, the AR device <NUM> may display other design information <NUM> on the viewport <NUM>. For example, the design information <NUM> may include information indicating whether the products in the BOM <NUM> are compliant with one or more codes and/or guidelines such as those described above. The AR device <NUM> may display design information <NUM> in response to the user input provided using the BOM validation input interface <NUM>. Alternatively or in addition, the AR device <NUM> may display design information <NUM> in response to the generation of the BOM <NUM> as described above. In some examples, the AR device <NUM> or via communication with a cloud sever having access to inventory information, may display whether or not one or more products in the BOM (e.g., the BOM <NUM>) are available for purchase or an estimate of when the one or more products may be available for purchase or delivery.

In some examples, the design information <NUM> may include suggestions of additional and/or replacement products. For example, the design information <NUM> may suggest one or more load control devices (e.g., relays, etc.) based on the number lighting fixtures and IoT devices included in the BOM <NUM> and the power ratings of the lighting fixtures and IoT devices. As another example, the design information <NUM> may suggest one or more replacement lighting fixtures to meet light level guidelines and/or requirements, occupancy-based lighting control requirements, plenum rating requirements, power density requirements, etc. In some example embodiments, the design information <NUM> may provide information indicating wire gauge recommendations based the number of lighting fixtures and load control devices included in the BOM <NUM>. A user may use the product menu <NUM> to add products to the BOM <NUM> or to replace products included in the BOM <NUM>.

In some examples, the user may order the products included in the BOM <NUM> using the order input interface <NUM>. For example, clicking or tapping the order input interface <NUM> may result in the e-commerce interface <NUM> or another page/interface being displayed on the viewport <NUM> for the execution of a purchase/ordering of the products included in the BOM.

In general, the AR device may execute software code included in the AR application or interfaced with the AR application to perform the operations described herein. Alternatively or in addition, the AR device <NUM> may send relevant information to another device (e.g., a cloud server) to perform some of the operations.

In some alternative examples, the BOM <NUM>, interfaces, etc. shown in <FIG> may be displayed in a different format, on different pages, etc.. In some alternative examples, one or more of the interfaces and information shown in <FIG> may be omitted.

<FIG> illustrate lighting design stages using the AR device <NUM> of <FIG> according to another example embodiment. Although the descriptions below are presented generally with respect to the AR device <NUM> of <FIG>, the description is equally applicable to the AR devices <NUM>, <NUM> of <FIG>. In some example embodiments, <FIG> illustrates a real-time image <NUM> of a target area displayed on the viewport <NUM> of the AR device <NUM> incorporating the lighting design AR application. <FIG> illustrates a modified image <NUM> of a target area along with lighting fixture <NUM>-D models <NUM> displayed for selection by a user. For example, as described with respect to <FIG>, a user may provide an input to the AR device <NUM> (e.g., via the input area <NUM> or via the viewport <NUM>) to apply a darkening filter to the pixels of the viewport <NUM> such that the modified image <NUM> is a result of the real-time image <NUM> and the darkening of the viewport <NUM>. As can be seen in <FIG>, the real-time image <NUM> of the target area may still be visible to the user after the darkening filter is applied to allow the user to place one or more selected lighting fixture <NUM>-D models at a desired location with respect to the real-time image <NUM> of the target area.

In some example embodiments, <FIG> illustrates an image <NUM> that includes a real-time image <NUM> of the target area overlaid with a lighting pattern associated with or resulting from a selected lighting fixture <NUM>-D model <NUM> and an associated photometric file. The AR device <NUM> executes the lighting design AR application to process the photometric data of the selected <NUM>-D model <NUM> and generate the lighting pattern, which is overlaid on the real-time image <NUM> of the target area along with the selected lighting fixture <NUM>-D model <NUM>. For example, the AR device <NUM> may use the photometric data associated with the selected lighting fixture <NUM>-D model <NUM> to generate a display model (e.g., a polygon or another display model) representing the lighting pattern, and the generated display model may be displayed on the viewport <NUM> overlaid on the real-time image <NUM> of the target area. In some alternative embodiments, the display model may be received/retrieved by the AR device <NUM> in response to the selection of the lighting fixture <NUM>-D model <NUM>. For example, the AR device <NUM> may receive/retrieve the display model representing the lighting pattern (e.g., as a polygon or another type of display model) from another device (e.g., a local or remote server) that generates the display model after receiving from the AR device <NUM> information indicating the selected lighting fixture <NUM>-D model <NUM>.

Information such as color temperature, luminance levels, etc. contained in the photometric data may be represented by the parameters of the display model, and the pixels of the viewport <NUM> are changed/set based on the parameters of the display model. For example, different points or parts of a polygon (or another display model) may be associated with different luminance levels, color temperature values, etc. contained in the photometric data associated with the selected lighting fixture <NUM>-D model <NUM>. The AR device <NUM> may display the real-time image of the target area overlaid with the polygon by adjusting/setting the pixels of the viewport <NUM> to account for the parameters of the polygon.

In some example embodiments, the AR device <NUM> may use the photometric data associated with the selected lighting fixture <NUM>-D model <NUM> along with the lighting conditions in the target area to generate a polygon (or another display model) that has parameters that are based on both the photometric data and the lighting conditions. The AR device <NUM> uses the lighting condition sensed by the ambient light sensor <NUM> to generate the parameters of a display model. In some example embodiments, the AR device <NUM> may generate a display model based on the photometric data of the selected lighting fixture <NUM>-D model <NUM> and modify the parameters of the display model based on the sensed lighting condition.

In some example embodiments, the AR device <NUM> may execute an artificial intelligence application to determine objects and structures in the target area, for example, based on the real-time imager of the target area. For example, the AR device <NUM> may identify reflective surfaces, walls, furniture, etc. and account for reflections, shadows, etc. in generating the display model that is overlaid on the real-time image displayed on the viewport <NUM>.

The AR device <NUM> executes the lighting design AR application to selectively remove/change the darkening filter applied to the pixels of the viewport <NUM> as described above with respect to <FIG>. A shown in <FIG> an area <NUM> may be well lit as compared to areas <NUM> and <NUM> that may be dimly lit. The lighting design AR application may process the photometric data of the selected <NUM>-D model <NUM> to determine the appearance of shadows, etc., resulting in realistic lighting patterns.

As illustrated in <FIG>, the selected lighting fixture <NUM>-D model <NUM> is displayed in the real-time image of the target area, enabling the user to assess how the corresponding lighting fixture will look when installed in the target area. Thus, a user, such as a lighting designer, owner, etc., may more effectively perform lighting design of a particular area (e.g., a living room, a bedroom, a hallway, etc.) without having to install actual lighting fixtures while minimizing design errors.

As described above, a display model that represents the photometric data associated with one or more lighting fixtures may be a 2D polygon, a <NUM>-D polygon, a combination of 2D and/or <NUM>-D polygons, graphical image(s), another type of image(s), etc. A polygon as an example of a display model may be a 2D polygon, a <NUM>-D polygon, a combination of 2D and/or <NUM>-D polygons, graphical image(s), another type of image(s), etc..

<FIG> illustrates a lighting characteristic selector <NUM> of the augmented reality device of <FIG> according to an example embodiment. In some example embodiments, the lighting characteristic selector <NUM> may be used to change and/or select a color temperature or even a particular color (e.g., wavelength) of the light from the <NUM>-D model <NUM>. For example, the selector <NUM> may be used to select from discrete color temperature values corresponding to lighting fixture <NUM>-D models stored or available to the AR device <NUM>. Alternatively or in addition, the selector <NUM> or another selector may be used to change and/or select luminance levels, light output patterns of the selected <NUM>-D model <NUM>, etc. In some example embodiments, changing a characteristic of the light from the selected <NUM>-D model <NUM> may effectively result in a replacement of the selected <NUM>-D model by another <NUM>-D model or in the replacement of the associated photometric file(s), for example, when the outward appearance of the <NUM>-D model is not affected by the changes. The change/selection of a characteristic of the light from the selected <NUM>-D model <NUM> may result in a corresponding change being reflected in the image <NUM> displayed on the viewport <NUM>.

<FIG> illustrate the lighting pattern of <FIG> and <FIG> with different color temperatures according to an example embodiment. The lighting pattern with the particular color temperature shown in <FIG> may be produced by the AR device based on a user selection of a respective color temperature using, for example, the lighting characteristics selector <NUM> of <FIG> or another selection means. For example, the CCT of <NUM> shown in <FIG> may correspond to the bottom most position of the lighting characteristics selector <NUM>, the CCT of <NUM> shown in <FIG> may correspond to a middle position of the lighting characteristics selector <NUM>, and the CCT of <NUM> shown in <FIG> may correspond to the upper most position of the lighting characteristics selector <NUM>. To illustrate, in response to a CCT selection indicated by the lighting characteristics selector <NUM>, the AR devices <NUM> may execute code to change the color (and as needed, the brightness) of relevant pixels. For example, subpixels of each relevant pixel may be adjusted to produce a desired color of the particular pixel. In some example embodiments, the AR device <NUM> applies a darkening, brightening, and/or color adjustment filter to the pixels of the viewport <NUM> to display the lighting pattern with the CCT corresponding to the selection indicated by the lighting characteristics selector <NUM>.

In some example embodiments, particular positions of the lighting characteristics selector <NUM> may be associated with a respective display model stored in or otherwise retrievable by the AR device <NUM>. For example, each model may be a polygon that has a shape corresponding to a particular light distribution pattern, where the polygon has display parameters corresponding to a CCT value, etc. To illustrate, the AR device <NUM> may modify the pixels of the viewport <NUM> to display the polygon (i.e., the display model) overlaid on the real-time image of the target area. In some example embodiments, the AR device <NUM> may generate or retrieve the CCT related parameters of the polygon based on the CCT indicated by the lighting characteristics selector <NUM>. In some example embodiments, the AR device <NUM> may generate or modify the parameters of the polygon based on the CCT selection indicated by the lighting characteristics selector <NUM> along with the lighting condition in the target area, for example, sensed by the ambient light sensor <NUM> of the device <NUM>.

In some alternative embodiments, each color temperature of the lighting pattern shown in <FIG> may be produced by selecting a lighting fixture <NUM>-D model from among the <NUM>-D models <NUM> shown in <FIG>, where the selected <NUM>-D model is associated with a photometric file that reflects the desired color temperature.

In some example embodiments, the lighting pattern of <FIG> and <FIG> may also be produced with a desired luminance level in a similar manner as described with color temperature. To illustrate, the lighting pattern of <FIG> and <FIG> may also be produced with a desired luminance level by selecting the desired luminance level using the selector <NUM> of <FIG> or another means. For example, a top position of the lighting characteristics selector <NUM> may correspond to a first luminance level (e.g., <NUM> lumens), a middle position of the lighting characteristics selector <NUM> may correspond to a second luminance level (e.g., <NUM> lumens), and a bottom position of the lighting characteristics selector <NUM> may correspond to a third luminance level (e.g., <NUM> lumens). To illustrate, in response to a luminance level selection indicated by the lighting characteristics selector <NUM>, the AR devices <NUM> may execute code to change the color (and as needed, the brightness) of relevant pixels. In some example embodiments, the AR device <NUM> applies a darkening or brightening filter to the pixels of the viewport <NUM> to display the lighting pattern with the luminance levels corresponding to the selection indicated by the lighting characteristics selector <NUM>.

In some example embodiments, particular positions of the lighting characteristics selector <NUM> may be associated with a respective display model stored in or otherwise retrievable by the AR device <NUM>. For example, each display model may be a polygon that has a shape corresponding to a particular light distribution pattern, where the polygon has display parameters corresponding to luminance levels, etc. To illustrate, the AR device <NUM> may modify the pixels of the viewport <NUM> to display the polygon (i.e., the display model) overlaid on the real-time image of the target area. In some example embodiments, the AR device <NUM> may generate or retrieve the luminance level related parameters of the polygon based on the luminance level indicated by the lighting characteristics selector <NUM>. In some example embodiments, the AR device <NUM> may generate or modify the parameters of the polygon based on the luminance level selection indicated by the lighting characteristics selector <NUM> along with the lighting condition in the target area, for example, sensed by the ambient light sensor <NUM> of the device <NUM>.

Alternatively, the desired luminance intensity may be achieved by selecting a <NUM>-D model associated with a photometric file that includes the desired luminance intensity.

<FIG> illustrates an alternative lighting pattern produced by the AR device <NUM> according to an example embodiment. The alternative lighting pattern may result from the selection of a lighting fixture <NUM>-D model <NUM> that produces a different lighting pattern as compared to the <NUM>-D model <NUM> of <FIG>.

In some example embodiments, the color temperature, luminance intensity, lighting pattern, and/or another characteristic of the light from a lighting fixture <NUM>-D model may be changed after the initial lighting pattern as shown in <FIG> (also in <FIG>) is displayed.

<FIG> illustrates an image frame <NUM> screenshots of images produced using the augmented reality device of <FIG> according to an example embodiment. These images may also be displayed on a screen of the AR devices <NUM>, <NUM> of <FIG>. In some example embodiments, going from left to right of each row of images, in the top row, the leftmost image shows an icon (LiAR) for activating the AR application described above. The next image in the top row shows a menu that can be used, for example, to select a lighting fixture model to be placed in a real-time image of the area viewable by the camera of the AR device. The right two images on the top row and all the images in the middle row show <NUM>-D models of different lighting fixtures overlaid on the image of the physical area (e.g., an office hallway) as viewed by the camera of the AR device.

The images on the bottom row show <NUM>-D models of different outdoor lighting fixtures overlaid on the image of the physical area (e.g., a walkway) as viewed by the camera of the AR device. In general, the AR devices <NUM>, <NUM>, <NUM> may execute AR lighting design application to overlay one or more <NUM>-D models of indoor and outdoor lighting fixtures on images of physical spaces (e.g., indoor space such as living room, kitchen, hallway, halls, etc. and outdoor spaces such as parking garages, open parking lots, walkways, stadiums, auditoriums, etc. to make a realistic assessment of the appearance of the lighting fixtures as well as the lighting effects of the lighting fixtures prior to the installation of the lighting fixtures.

<FIG> illustrates a <NUM>-D model <NUM> of a lighting fixture with an integrated sensor <NUM> according to an example embodiment. In some example embodiments, the AR devices <NUM>, <NUM>, <NUM> may be used in IoT design (e.g., placement of IoT devices in a space) in addition to or instead of the lighting design. The AR devices <NUM>, <NUM>, <NUM> may be used to perform IoT design in a similar manner as described above with respect to the AR device <NUM> and lighting design. For example, a menu <NUM> of <NUM>-D models of lighting fixtures with and without integrated IoT devices (e.g., the sensor <NUM>) and standalone IoT devices may be displayed or otherwise provided on the viewport <NUM> of the AR device <NUM> and on the corresponding display screen of the AR devices <NUM> and <NUM>. Each <NUM>-D model may be associated with a parameter data file that includes data indicating the detection range/angles/field of view <NUM> of the sensor <NUM> (e.g., a motion sensor, carbon dioxide sensor, carbon monoxide sensor, smoke sensor, etc.). A user may select a <NUM>-D model such as the <NUM>-D model <NUM> of a lighting fixture with the sensor <NUM> and place the <NUM>-D model on a desired location on the real-time image of a target area as described above with respect to <FIG>. The AR devices <NUM>, <NUM>, <NUM> may execute the AR application to overlay on the real-time image of the target area a display model (e.g., a polygon or another display model) corresponding to the detection range/angle of the sensor <NUM> in a similar manner as described above with respect to lighting design. For example, the AR device <NUM>, <NUM>, <NUM> may generate the display model or retrieve an existing display model associated with the selected <NUM>-D model <NUM>. In some example embodiments, the parameter data file may include other information that can be used to generate the display model without departing from the scope of this disclosure.

<FIG> illustrates a <NUM>-D model of a lighting fixture with an integrated camera <NUM> according to an example embodiment. In some example embodiments, a menu <NUM> of <NUM>-D models of lighting fixtures with and without integrated IoT devices (e.g., the camera <NUM>) and standalone IoT devices may be displayed or otherwise provided on the viewport <NUM> of the AR device <NUM> and on the corresponding display screen of the AR devices <NUM> and <NUM>. Each <NUM>-D model may be associated with a parameter data file that includes data indicating the field of view <NUM> of the camera <NUM>. A user may select a <NUM>-D model such as the <NUM>-D model <NUM> of a lighting fixture with the integrated camera <NUM> and place the <NUM>-D model on a desired location on the real-time image of a target area as described above with respect to <FIG>. The AR devices <NUM>, <NUM>, <NUM> may execute the AR application to overlay on the real-time image of the target area a display model (e.g., a polygon or another display model) corresponding to the field of view <NUM> of the camera <NUM> in a similar manner as described above with respect to lighting design. For example, the AR device <NUM>, <NUM>, <NUM> may generate the display model or retrieve an existing display model associated with the selected <NUM>-D model <NUM>. In some example embodiments, the parameter data file may include other information that can be used to generate the display model without departing from the scope of this disclosure.

<FIG> illustrates a <NUM>-D model of a lighting fixture with an integrated speaker or array of speakers <NUM> according to an example embodiment. In some example embodiments, a menu <NUM> of <NUM>-D models of lighting fixtures with and without integrated IoT devices (e.g., the speaker or array of speakers <NUM>) and standalone IoT devices may be displayed or otherwise provided on the viewport <NUM> of the AR device <NUM> and on the corresponding display screen of the AR devices <NUM> and <NUM>. Each <NUM>-D model may be associated with a parameter data file that includes data indicating the range and/or directionality <NUM> of the sound that can be produced by the speaker or array of speaker or array of speakers <NUM>, for example, at a maximum rating of the speaker or array of speakers <NUM> and/or at different percentages of the maximum rating of the speaker or array of speakers <NUM>. A user may select a <NUM>-D model such as the <NUM>-D model <NUM> of a lighting fixture with the integrated speaker or array of speakers <NUM> and place the <NUM>-D model on a desired location on the real-time image of a target area as described above with respect to <FIG>. The AR devices <NUM>, <NUM>, <NUM> may execute the AR application to overlay on the real-time image of the target area a display model (e.g., a polygon or another display model) corresponding to the range <NUM> of the speaker or array of speakers <NUM> in a similar manner as described above with respect to the photometric data in lighting design. For example, the AR device <NUM>, <NUM>, <NUM> may generate the display model or retrieve an existing display model associated with the selected <NUM>-D model <NUM>. In some example embodiments, the parameter data file may include other information that can be used to generate the display model without departing from the scope of this disclosure.

<FIG> illustrates a <NUM>-D model of a lighting fixture with an integrated microphone or array of microphones <NUM> according to an example embodiment. In some example embodiments, a menu <NUM> of <NUM>-D models of lighting fixtures with and without integrated IoT devices (e.g., the microphone or array of microphones <NUM>) and standalone IoT devices may be displayed or otherwise provided on the viewport <NUM> of the AR device <NUM> and on the corresponding display screen of the AR devices <NUM> and <NUM>. Each <NUM>-D model may be associated with a parameter data file that includes data indicating the range and/or directionality <NUM> that a sound generated at a particular decibel or different decibels can be detected by the microphone or array of microphones <NUM>. A user may select a <NUM>-D model such as the <NUM>-D model <NUM> of a lighting fixture with the integrated microphone or array of microphones <NUM> and place the <NUM>-D model on a desired location on the real-time image of a target area as described above with respect to <FIG>.

The AR devices <NUM>, <NUM>, <NUM> may execute the AR application to overlay on the real-time image of the target area a display model (e.g., a polygon or another display model) corresponding to the range <NUM> of the microphone or array of microphones <NUM> in a similar manner as described above with respect to the photometric data in lighting design. For example, the AR device <NUM>, <NUM>, <NUM> may generate the display model or retrieve an existing display model associated with the selected <NUM>-D model <NUM>. In some example embodiments, the parameter data file may include other information that can be used to generate the display model without departing from the scope of this disclosure.

In some example embodiments, the AR devices <NUM>, <NUM>, <NUM> and the AR application may be used to perform lighting as well as IoT design, where <NUM>-D models of lighting fixtures with and without IoT devices are presented to the user on the display screen of the AR devices. In general, operations provided herein with respect to one of the AR devices <NUM>, <NUM>, <NUM> are applicable to other ones of the AR devices <NUM>, <NUM>, <NUM>.

In some alternative embodiments, a parameter data file that includes alternative gradient of lighting information may be used instead of the photometric data file described above. The description provided herein with respect to photometric data and photometric data files may be equally applicable to parameter data and parameter data files with alternative gradient of lighting data.

<FIG> and <FIG> illustrate use of the augmented reality device <NUM> of <FIG> to simulate sensor-controlled lighting behavior according to an example embodiment. In some example embodiments, the AR devices <NUM>, <NUM> of <FIG> may be used instead of the AR device <NUM>. Referring to <FIG>, in some example embodiments, the AR device <NUM> may display a real-time image <NUM> of a target area, for example, a parking lot or garage or an indoor space in a similar manner as described above with respect to <FIG> and <FIG>. The AR device <NUM> may display one or more <NUM>-D models such as a <NUM>-D model <NUM> of a lighting fixture with one or more IoT devices shown as an IoT device <NUM>. For example, the <NUM>-D model <NUM> may correspond to the <NUM>-D model <NUM> shown in <FIG> or <NUM> shown in <FIG>.

In some example embodiments, the IoT device <NUM> may have an operational range <NUM>. For example, the IoT device <NUM> may be a sensor such as a motion sensor. To illustrate, the operational range <NUM> of the IoT device <NUM> may be the detection range, angle, or field of view of a motion sensor. As another example, the IoT device <NUM> may be a camera, where the operational range <NUM> of the IoT device <NUM> may be the field of view of the camera.

In some example embodiments, some operations of the lighting fixture represented by the <NUM>-D model <NUM> may depend on or may be controlled by the one or more IoT devices of the lighting fixtures. To illustrate, after the one or more <NUM>-D models, including the <NUM>-D model <NUM> that includes the IoT device <NUM>, are displayed on the viewport <NUM>, a user carrying the AR device may move toward the real-time image <NUM> and the IoT device <NUM> (i.e., toward the <NUM>-D model <NUM>). When the user reaches the operational range <NUM> (which may or may not be displayed in the viewport <NUM>) of the IoT device <NUM>, a lighting pattern <NUM> may be displayed by the AR device <NUM>. The display of the lighting pattern <NUM> in response to the user moving into or within the operational range <NUM> of the IoT device <NUM> simulates the behavior of the lighting fixture with one or more IoT devices represented by the <NUM>-D model <NUM> in response to a person (or a car or other object detected by the IoT device) moving into or within the detection or sensing region of the one or more IoT devices.

In some example embodiments, the lighting pattern <NUM> may be removed from the viewport <NUM> in response to the user holding the AR device <NUM> moving out of the operational range <NUM> of the IoT device. For example, if the user returns to the original location in the target physical area, the image displayed on the viewport <NUM> may be similar to the image shown in <FIG>.

By simulating the behavior of lighting fixtures without installing the lighting fixtures and the IoT devices, a user may achieve desirable results, confirm desired operation with the need for physical installation, and/or avoid some design errors. For example, more accurate location and/or orientation of IoT devices integrated with lighting fixtures or external to lighting fixtures may be determined by simulating the behavior of lighting fixtures in response to the IoT devices.

In some alternative embodiments, the IoT device <NUM> may be external to the lighting fixture represented by the <NUM>-D model <NUM>. In some example embodiments, the behavior of multiple lighting fixtures in response to one or more IoT devices may be simulated in a similar manner. In some example embodiments, the lighting pattern <NUM> may be similar to the lighting pattern shown in <FIG> or <FIG>. In some example embodiments, similar simulation of the operation of devices (that are not light fixtures, such as automatic doors, shades, fans, thermostats, displays or other controllable devices) being controlled or in communication with IoT device(s) in response to the AR device entering or leaving the simulated range or pattern associated with an operating characteristic of an IoT device(s) may be displayed on the AR device.

<FIG> illustrates a method <NUM> of augmented reality-based lighting and IoT design according to an example embodiment. Referring to <FIG>, in some example embodiments, the method <NUM> includes, at step <NUM>, displaying, by an augmented reality device (e.g., the AR device <NUM>, <NUM>, <NUM>), a real-time image of a target physical area on a display screen of the AR device. For example, the AR device <NUM> may display the real-time image <NUM> of the target physical area <NUM> as viewed by the camera <NUM>.

At step <NUM>, the method <NUM> may include displaying, by the augmented reality device, a lighting fixture <NUM>-D model on the display screen in response to a user input, where the lighting fixture <NUM>-D model is overlaid on the real-time image of the target physical area. For example, the <NUM>-D model <NUM> and other <NUM>-D models may be overlaid on the real-time image <NUM>. To illustrate, the lighting fixture <NUM>-D model may be overlaid on the real-time image <NUM> before or after a darkening filter has been applied to the real-time image <NUM> as described with respect to <FIG>. As another example, one or more lighting fixture <NUM>-D models may be overlaid on the real-time image <NUM> shown in <FIG>.

At step <NUM>, the method <NUM> may include displaying, by the augmented reality device, a lighting pattern on the display screen overlaid on the real-time image of the target physical area, where the lighting pattern is generated based on at least photometric data associated with the lighting fixture <NUM>-D model. For example, image <NUM>, including the lighting pattern, shown in <FIG> may be displayed by the AR device <NUM> by overlaying the lighting pattern on the real-time image <NUM> of the target physical area <NUM>. The AR device <NUM>, <NUM>, <NUM> or another device may generate the lighting pattern from at least the photometric data associated with the lighting fixture <NUM>-D model as described above.

In some example embodiments, the method <NUM> may include darkening the display screen before displaying the lighting fixture <NUM>-D model on the display screen as described with respect to <FIG> and <FIG>. For example, the real-time image of the target physical area may remain visible after the darkening of the display screen to allow the placement of lighting fixture <NUM>-D models at desired locations in the real-time image <NUM>. In some alternative embodiments, the darkening the display screen may be omitted when, for example, assessment of lighting pattern is not performed. For example, a lighting design may be focused on the aesthetic features of the lighting fixture(s) (or one or more of the light fixtures subcomponents such as a trim, optic, or accessories) in the target area instead of lighting patterns.

In some example embodiments, the method <NUM> may include changing a color temperature associated with the lighting pattern displayed on the display screen. The color temperature may be changed in response to a user input. For example, the lighting characteristic selector <NUM> may be used to change and/or select a color temperature as described with respect to <FIG>. In some alternative embodiments, replacing the displayed <NUM>-D model by another <NUM>-D model may result in a different color temperature.

In some example embodiments, the method <NUM> may include changing a luminance level associated with the lighting pattern displayed on the display screen. The luminance level may be changed in response to a user input. For example, the lighting characteristic selector <NUM> may be used to change and/or select a luminance level as described with respect to <FIG>. In some alternative embodiments, replacing the displayed <NUM>-D model by another <NUM>-D model may result in a different luminance level.

In some example embodiments, the method <NUM> may include displaying, by the augmented reality device, luminance level values indicating luminance levels associated with the lighting pattern overlaid on the real-time image of the target physical area, for example, as described with respect to <FIG>. The method <NUM> may also include displaying, by the augmented reality device, one or more other lighting fixture <NUM>-D models on the display screen in response to a user input, for example, as described with respect to <FIG>. The one or more lighting fixture <NUM>-D models may be overlaid on the real-time image of the target physical area in a similar manner as the <NUM>-D model at step <NUM>. In some alternative embodiments, one or more other lighting fixture <NUM>-D models may be added to the real-time image (e.g., the real-time image <NUM>, <NUM>) displayed on the display screen (e.g., the display screen <NUM>) before or after darkening the display screen. Alternatively, the darkening of the display screen may be omitted.

In some alternative embodiments, one or more steps of the method <NUM> may be omitted or may be performed in a different order than described above. Although some of the steps of the method <NUM> are described with respect to one or more images or figures, the steps may be applicable to other images and figures without departing from the scope of this disclosure. Although some of the steps of the method <NUM> are described with respect to the AR device <NUM>, the steps may be performed by the other AR devices including the AR device <NUM> and <NUM> without departing from the scope of this disclosure. In general, the steps of the method <NUM> may be performed by the AR devices <NUM>, <NUM>, <NUM>. For example, a controller (e.g., the controller <NUM>) of the AR devices may execute software code to perform the steps of the method <NUM>.

<FIG> illustrates a method <NUM> of augmented reality-based lighting and IoT design according to another example embodiment. Referring to <FIG>, in some example embodiments, the method <NUM> includes, at step <NUM>, displaying, by an augmented reality device (e.g., the AR device <NUM>, <NUM>, <NUM>), a real-time image of a target physical area on a display screen of the AR device. For example, the AR device <NUM> may display the real-time image <NUM> of the target physical area <NUM> as viewed by the camera <NUM>.

At step <NUM>, the method <NUM> may include displaying, by the augmented reality device, a <NUM>-D model of a lighting fixture with one or more IoT devices on the display screen in response to a user input, where the <NUM>-D model is overlaid on the real-time image of the target physical area. For example, the <NUM>-D model <NUM> may correspond to a lighting fixture with one or more integrated IoT devices (or, alternatively, one or more standalone IoT devices), and the <NUM>-D model <NUM> and other similar <NUM>-D models may be overlaid on the real-time image <NUM> shown in <FIG>. The <NUM>-D model <NUM> may be overlaid on the real-time image <NUM> before or after a darkening filter has been applied to the real-time image <NUM> as described with respect to <FIG>. As another example, a <NUM>-D model of a lighting fixture with one or more IoT devices may be overlaid on the real-time image <NUM> shown in <FIG>.

At step <NUM>, the method <NUM> may include displaying on the display screen, by the augmented reality device, a pattern overlaid on the real-time image of the target physical area, where the pattern corresponds to parameter data associated with the <NUM>-D model. For example, the pattern may correspond to the one or more operating characteristics associated with an IoT device(s) integrated with the lighting fixture correspond to the <NUM>-D model. In some example embodiments, a lighting pattern as described above, for example, with respect to <FIG> and <FIG> may also be overlaid on the real-time image displayed on the display screen.

To illustrate with an example, the one or more IoT devices may include one or more sensors, and the pattern overlaid on the real-time image may show the detection range, angle, and/or field of view of the one or more sensors. For example, a pattern showing the detection range/angle <NUM> shown in <FIG> may be overlaid on the real-time image <NUM> or <NUM> in a similar manner as the lighting pattern or luminance levels described above. The <NUM>-D model of the lighting fixture with the one or more IoT devices may be associated with a parameter data file that includes data indicating the detection range, angle, and/or field of view of the one or more sensors.

As another example, the one or more IoT devices may include one or more cameras, and the pattern overlaid on the real-time image may show the field of view of the one or more cameras. For example, a pattern showing the field of view <NUM> of the camera <NUM> shown in <FIG> may be overlaid on the real-time image <NUM> or <NUM> in a similar manner as the lighting pattern or luminance levels described above. The <NUM>-D model of the lighting fixture with the one or more IoT devices may be associated with a parameter data file that includes data indicating the field of view of the one or more cameras.

As another example, the one or more IoT devices may include one or more speakers, and the pattern overlaid on the real-time image may show the range and/or directionality of a sound produced by the one or more speakers, for example, at a particular decibel (a decibel value or values may also be displayed). For example, a pattern showing the range and/or directionality <NUM> of the speaker <NUM> shown in <FIG> may be overlaid on the real-time image <NUM> or <NUM> in a similar manner as the lighting pattern or luminance levels described above. The <NUM>-D model of the lighting fixture with the one or more IoT devices may be associated with a parameter data file that includes data indicating the range and/or directionality of a sound produced by the one or more speakers at one or more decibels.

As another example, the one or more IoT devices may include one or more microphones, and the pattern overlaid on the real-time image may show the sound detection range and/or directionality of the one or more microphones, for example, at a particular decibel (a decibel value or values may also be displayed). For example, a pattern showing sound detection range and directionality <NUM> of the microphone <NUM> shown in <FIG> may be overlaid on the real-time image <NUM> or <NUM> in a similar manner as the lighting pattern or luminance levels described above. The <NUM>-D model of the lighting fixture with the one or more IoT devices may be associated with a parameter data file that includes data indicating sound detection range and/or directionality of the one or more microphones.

In some example embodiments, a lighting pattern as described above, for example, with respect to <FIG> and <FIG> may also be overlaid on the real-time image displayed on the display screen.

In some example embodiments, one or more steps of the method <NUM> may be performed using <NUM>-D models of standalone IoT devices. In some example embodiments, one or more steps of the method <NUM> may be performed as one or more steps of the method <NUM> without departing from the scope of this disclosure. In some alternative embodiments, one or more steps of the method <NUM> may be omitted or may be performed in a different order than described above. Although some of the steps of the method <NUM> are described with respect to one or more images or figures, the steps may be applicable to other images and figures without departing from the scope of this disclosure. In general, the steps of the method <NUM> may be performed by the AR devices <NUM>, <NUM>, <NUM>. For example, a controller (e.g., the controller <NUM>) of the AR devices may execute software code to perform the steps of the method <NUM>.

<FIG> illustrates a method <NUM> of augmented reality-based lighting and IoT design according to another example. Referring to <FIG>, in some example embodiments, the method <NUM> includes, at step <NUM>, displaying, by an augmented reality device (e.g., the AR device <NUM>, <NUM>, <NUM>), a real-time image of a target physical area on a display screen of the AR device. For example, the AR device <NUM> may display the real-time image <NUM> of the target physical area <NUM> as viewed by the camera <NUM>. As another example, the AR device <NUM> may display the real-time image <NUM> of a target physical area as viewed by the camera <NUM>.

At step <NUM>, the method <NUM> may include displaying, by the augmented reality device, a lighting fixture <NUM>-D model on the display screen in response to a user input, where the lighting fixture <NUM>-D model is overlaid on the real-time image of the target physical area. For example, the <NUM>-D model <NUM> may correspond to a lighting fixture with or without one or more integrated IoT devices, and the <NUM>-D model <NUM> and other similar <NUM>-D models may be overlaid on the real-time image <NUM> shown in <FIG>. As another example, a <NUM>-D model of a lighting fixture with or without one or more IoT devices may be overlaid on the real-time image <NUM> shown in <FIG>.

At step <NUM>, the method <NUM> may include generating, by the augmented reality device, a BOM (or purchase order) that includes a lighting fixture corresponding to the lighting fixture <NUM>-D model. For example, the AR device <NUM> may generate the BOM <NUM> shown in <FIG>. To illustrate, the AR device <NUM> may generate or retrieve from a remote device (e.g., cloud server) that generates the BOM <NUM> in response to a user input provided to the AR device <NUM>, for example, via the BOM generation input interface <NUM> or another input interface. Alternatively, the BOM may be generated upon a completion of lighting and IoT design process that may be indicated in one of several means as can be understood by those of ordinary skill in the art with the benefit of this disclosure.

For example, the <NUM>-D model <NUM> and other <NUM>-D models may be overlaid on the real-time image <NUM>. To illustrate, the lighting fixture <NUM>-D model may be overlaid on the real-time image <NUM> before or after a darkening filter has been applied to the real-time image <NUM> as described with respect to <FIG>. As another example, one or more lighting fixture <NUM>-D models may be overlaid on the real-time image <NUM> shown in <FIG>.

In some examples, the method <NUM> may include displaying, by the augmented reality device, a lighting pattern on the display screen overlaid on the real-time image of the target physical area, for example, as described with respect to the method <NUM>. In some examples, the method <NUM> may include displaying, by the augmented reality device, a product menu (e.g., the product menu <NUM> and/or a search bar to search for products) on the display screen (e.g., the viewport <NUM>) for use by a user to add one or more products to the BOM, such as the BOM <NUM>.

In some examples, the method <NUM> may include displaying, by the augmented reality device, a message (e.g., the design information <NUM>) suggesting one or more other lighting products to be added to the BOM (e.g., the BOM <NUM>). In some examples, the method <NUM> may include determining, by the augmented reality device or via communication with a cloud sever, whether one or more products in the BOM (e.g., the BOM <NUM>) are available for purchase or an estimate of when the one or more products may be available for purchase or delivery. In some examples, the method <NUM> may include determining, by the augmented reality device or via communication with a cloud sever, whether one or more products in the BOM (e.g., the BOM <NUM>) are compliant with an electrical or lighting code or guideline (e.g., ICC, OSHA, Title <NUM> of the California Code of Regulations, and/or other codes or standards). In some examples, the method <NUM> may include displaying, by the augmented reality device, a message e.g., the design information <NUM>) indicating whether the one or more products in the BOM are compliant with the electrical or lighting code or guideline. The displayed information (e.g., the design information <NUM>) may also include another message displayed by the AR device suggesting one or more other lighting products as replacements to one or more products included in the BOM. In some examples, the method <NUM> may also include displaying a message indicating whether one or more lighting fixtures listed in the BOM provide a light having a lighting level that is compliant with an electrical or lighting code or guideline. For example, the message may be included in the design information <NUM> displayed on the viewport <NUM>.

In some examples, one or more steps of the method <NUM> may be performed as one or more steps of the methods <NUM> and <NUM>. In some alternative examples, one or more steps of the method <NUM> may be omitted or may be performed in a different order than described above. Although some of the steps of the method <NUM> are described with respect to one or more images or figures, the steps may be applicable to other images and figures. In general, the steps of the method <NUM> may be performed by the AR devices <NUM>, <NUM>, <NUM>. For example, a controller (e.g., the controller <NUM>) of the AR devices may execute software code to perform the steps of the method <NUM>.

<FIG> illustrates a method of augmented reality-based lighting and IoT design according to another example embodiment. Referring to <FIG>, in some example embodiments, the method <NUM> includes, at step <NUM>, displaying, by an augmented reality device (e.g., the AR device <NUM>, <NUM>, <NUM>), a real-time image of a target physical area on a display screen of the AR device. For example, the AR device <NUM> may display the real-time image <NUM> of the target physical area <NUM> as viewed by the camera <NUM>. As another example, the AR device <NUM> may display the real-time image <NUM> of a target physical area as viewed by the camera <NUM>.

At step <NUM>, the method <NUM> may include displaying, by the augmented reality device, a <NUM>-D model of a lighting fixture with one or more IoT devices on the display screen in response to a user input, where the <NUM>-D model is overlaid on the real-time image of the target physical area. For example, the <NUM>-D model <NUM> may correspond to a lighting fixture (or other device controlled or in communication with one or more IoT devices) with or without one or more integrated IoT devices, and the <NUM>-D model <NUM> and other similar <NUM>-D models may be overlaid on the real-time image <NUM> shown in <FIG>. As another example, a <NUM>-D model of a lighting fixture with or without one or more IoT devices may be overlaid on the real-time image <NUM> shown in <FIG>.

At step <NUM>, the method <NUM> may include displaying on the display screen, by the augmented reality device, a lighting pattern overlaid on the real-time image of the target physical area in response to the augmented reality device moving within an operational range of the one or more IoT devices. For example, the lighting pattern may be similar to the lighting pattern shown in <FIG> and <FIG> or may be intended to just show that the <NUM>-D model is emitting a light similar to the emitted light shown in <FIG>. Alternatively, the lighting pattern may have a different appearance than shown in <FIG>, <FIG>, and <FIG> without departing from the scope of this disclosure. In some example embodiments, the method <NUM> may include, before displaying the light pattern, displaying on the display screen an IoT device pattern overlaid on the real-time image of the target physical area in response to the augmented reality device moving within an operational range of the one or more IoT devices. For example, the IoT device pattern may correspond to the operational range <NUM> (e.g., range, angles, field of view, etc.) of the one or more IoT devices.

In some example embodiments, the method <NUM> may include displaying on the display screen, by the augmented reality device, an IoT device pattern overlaid on the real-time image of the target physical area, for example, as shown in <FIG>. For example, the one or more IoT devices may include one or more motion sensors, where the operational range of the one or more IoT devices is a detection range of the one or more motion sensors such as the detection range/angles/field of view <NUM> of the sensor <NUM> shown in <FIG> or the operational range <NUM> (e.g., detection range) of the one or more IoT devices <NUM> (e.g., one or more sensors) shown in <FIG>. As another example, the one or more IoT devices may include one or more cameras, where the operational range of the one or more IoT devices is a field of view of the one or more cameras such as the field of view <NUM> of the camera <NUM> shown in <FIG> or the operational range <NUM> (e.g., field of view) of the one or more IoT devices <NUM> (e.g., one or more cameras) shown in <FIG>.

In some example embodiments, the method <NUM> includes removing the overlaid lighting pattern from the display screen in response to the AR device moving out of the operational range of the one or more IoT devices. For example, when a person carrying the AR device <NUM> moves outside of the operational range <NUM> of the one or more loT devices (e.g., one or more motion sensors and/or cameras), the light pattern illustrated in <FIG> may be removed such that, when the AR device <NUM> is returned to the same position as in <FIG>, the image displayed on the AR device <NUM> appears similar to the image shown in <FIG>.

In some example embodiments, one or more steps of the method <NUM> may be performed to simulate the operation of devices (that are not light fixtures, such as automatic doors, shades, fans, thermostats, displays or other controllable devices) being controlled or in communication with IoT device(s) in response to the AR device entering or leaving the simulated range or pattern associated with an operating characteristic of an IoT device. In some example embodiments, one or more steps of the method <NUM> may be performed as one or more steps of the other methods described herein without departing from the scope of this disclosure. In some alternative embodiments, one or more steps of the method <NUM> may be omitted or may be performed in a different order than described above. Although some of the steps of the method <NUM> are described with respect to one or more images or figures, the steps may be applicable to other images and figures without departing from the scope of this disclosure. In general, the steps of the method <NUM> may be performed by the AR devices <NUM>, <NUM>, <NUM>. For example, a controller (e.g., the controller <NUM>) of the AR devices may execute software code to perform the steps of the method <NUM>.

In the above description, a display model that represents photometric data or other parameter data associated with one or more lighting fixtures or parameter data associated with one or more IoT devices may be a 2D polygon, a <NUM>-D polygon, a combination of 2D and/or <NUM>-D polygons, graphical image(s), another type of image(s), etc. A polygon as an example of a display model may be a 2D polygon, a <NUM>-D polygon, a combination of 2D and/or <NUM>-D polygons, graphical image(s), another type of image(s), etc..

Claim 1:
An augmented reality-based lighting design method, comprising:
displaying, by an augmented reality device (<NUM>, <NUM>, <NUM>), a real-time image of a target physical area on a display screen (<NUM>);
displaying, by the augmented reality device (<NUM>, <NUM>, <NUM>), a lighting fixture <NUM>-D model (<NUM>-<NUM>, <NUM>) on the display screen (<NUM>) in response to a user input, wherein the lighting fixture <NUM>-D model is overlaid on the real-time image of the target physical area; the method characterized by
sensing, by an ambient light sensor (<NUM>) comprised in the augmented reality device (<NUM>, <NUM>, <NUM>), ambient lighting conditions in the target physical area;
generating, by the augmented reality device (<NUM>, <NUM>, <NUM>), a display model based on at least photometric data associated with the lighting fixture <NUM>-D model (<NUM>-<NUM>, <NUM>), the display model representing a lighting pattern (<NUM>) resulting from the lighting fixture <NUM>-D model (<NUM>-<NUM>, <NUM>); and
displaying, by the augmented reality device (<NUM>, <NUM>, <NUM>), the lighting pattern on the display screen overlaid on the real-time image of the target physical area,
wherein the lighting pattern (<NUM>) displayed on the display screen (<NUM>) accounts for the sensed ambient lighting conditions in the target physical area.