Graphical user interface for specifying 3D position

A graphical user interface (GUI) can specify a three-dimensional position. The GUI can include a puck that is user-positionable in two dimensions. A distance between the puck and a reference point on the graphical user interface can determine a radius of the three-dimensional position. An azimuthal orientation of the puck with respect to the reference point can determine an azimuthal angle of the three-dimensional position. The GUI can include an elevation controller, separate from the puck, to specify an elevation of the three-dimensional position. The elevation controller can specify an elevation angle, which can determine a position along the surface of a virtual sphere as a function of elevation angle. In addition, or alternatively, the elevation controller can specify an elevation height, which can determine a position along a line, such as directly upward or downward, as a function of elevation height.

FIELD OF THE DISCLOSURE

The technology described in this document relates generally to a two-dimensional (2D) graphical user interface (GUI) used to specify three-dimensional (3D) positions of virtual objects, such as audio objects or audio signals.

BACKGROUND OF THE DISCLOSURE

Many applications require specifying a three-dimensional position using a two-dimensional graphical user interface. For example, mixing sound, such as for a cinematic production or a video game, can require specifying a three-dimensional position of a sound source. As a specific example, a sound of an airplane flying overhead can involve specifying a position for the airplane and positioning a sound of an airplane engine at the specified position.

Specifying the position can be difficult or impossible with existing two-dimensional graphical user interfaces. For example, it may not be possible to dynamically specify a position of a (virtual) airplane as it flies by at a constant height, using existing two-dimensional graphical user interfaces.

As a result, there exists a need for an improved graphical user interface, which can specify a position of a (virtual) object as the object moves through (virtual) space in three dimensions.

Corresponding reference characters indicate corresponding parts throughout the several views. Elements in the drawings are not necessarily drawn to scale. The configurations shown in the drawings are merely examples and should not be construed as limiting the scope of the invention in any manner.

DETAILED DESCRIPTION

A graphical user interface (GUI) can specify a three-dimensional position. The GUI can include a puck that is user-positionable in two dimensions. A distance between the puck and a reference point on the graphical user interface can determine a radius of the three-dimensional position. An azimuthal orientation of the puck with respect to the reference point can determine an azimuthal angle of the three-dimensional position. The GUI can include an elevation controller, separate from the puck, to specify an elevation of the three-dimensional position. The elevation controller can specify an elevation angle, which can determine a position along the surface of a virtual sphere as a function of elevation angle. In addition, or alternatively, the elevation controller can specify an elevation height, which can determine a position along a line, such as directly upward or downward, as a function of elevation height.

In a specific example, when a radial lock mode is selected, user-initiated movement of the puck and/or elevation angle control allows specification of the three-dimensional position only around the surface of a virtual sphere with a constant radius from the reference point. Such a radial lock mode can be useful for the specific example of objects such as special effects orbiting the observer. In this specific example, because the special effects are at a constant radius away from the observer, the special effects can retain a constant volume level as they orbit the observer. Using existing graphical user interfaces with Cartesian controls, it can be cumbersome or impossible to create these constant-radius orbits.

FIG. 1shows an example of a system100that uses a graphical user interface102to specify a three-dimensional position104, in accordance with some embodiments. In the example ofFIG. 1, the system100can produce a mix of one or more audio signals to account for three-dimensional placement of the audio signals. In other examples, other types of systems, including systems that postpone audio mixing as a downstream operation or do not use audio, can use the graphical user interface102, as discussed below, to specify a three-dimensional position104.

A processor106can execute instructions stored in memory116, such that when the instructions are executed on the processor106, the instructions cause the processor106to generate the graphical user interface102and present the graphical user interface102on a display118to a user. The graphical user interface102can accept input from the user through an input device120such as a mouse or track-ball, a keyboard, and/or a touch-sensitive screen, which can optionally be the display. Through the graphical user interface102, the user can reposition user-positionable elements in the graphical user interface102to desired positions, such as by clicking on the elements, or dragging the elements to the desired positions. Through the graphical user interface102, the user can enter particular numerical values, such as by highlighting a direct entry box on the graphical user interface and entering a numerical value through a keyboard, a touch pad, or a virtual keyboard shown on the touch-sensitive display.

In some examples, the system100can operate as an authoring tool, which can trace a time-varying path of one or more virtual objects over time. When used as an authoring tool, the system100can specify three-dimensional positions, and times at which the virtual object is present at the specified positions and can save the specified positions and times in random access memory (RAM) or one or more files on a storage device114coupled to the processor106. The storage device114can be a local storage device or can be coupled via a wired or wireless network to a server.

In some examples, the system100can operate as an automated or rendering tool, which can retrieve a time-varying path of one or more virtual objects over time. When used as an automated rendering tool, the system100can apply three-dimensional positions in random access memory (RAM) or one or more files on a storage device114coupled to the processor106.

A spatial mixing engine110can receive the three-dimensional position(s)104, either directly from the processor106or from one or more files saved on the storage device coupled to the processor106. The spatial mixing engine110can process one or more source audio signals108, by placing each signal108at the corresponding specified three-dimensional position104, and simulating propagation of the signals from the specified positions to a listener. In some examples, the listener can be positioned at an origin of the three-dimensional coordinate system. In some examples, the listener can be a single point, positioned at the origin. In other examples, the listener can be simulated as having two ears on opposite sides of the origin. In some examples, the listener can have a fixed orientation, with respect to the three-dimensional coordinate system. In some examples. In other examples, such as for virtual reality or augmented reality video games, the listener can have a time-varying orientation, with respect to the three-dimensional coordinate system, depending on a real-time orientation of a headset or a head tracker or eye-tracker.

In some examples, such as the configuration ofFIG. 1, the graphical user interface102and the spatial mixing engine110can be coupled to the same processor106. For example, the processor106can run a self-contained audio mixing suite, in which the graphical user interface102specified positions104, and the spatial mixing engine uses those positions to perform the audio mixing and generate mixed audio signals112.

In some examples, the graphical user interface102can be coupled to a first processor, and the spatial mixing engine110can be coupled to a second processor. For example, the first processor can run software that specifies the positions of the audio elements, and the second processor can run software that performs the spatial mixing of the audio signals at the specified positions. Other configurations are also possible.

In some examples, the processor106can display a graphical user interface102, can receive a user-specified three-dimensional position through the graphical user interface102, and can store data representing the specified three-dimensional position on a storage device114coupled to the processor106. The graphical user interface102is shown in detail inFIG. 4and is discussed below.

FIG. 2shows an example of a three-dimensional position200, in accordance with some embodiments. In general, a three-dimensional position200can be specified by three coordinates in a specified coordinate system. The system ofFIG. 1can optionally generate one or more three-dimensional positions, and save them to a storage device, optionally with suitable saved times at which a virtual object is intended to be present at the saved positions.

FIG. 3shows two examples of three-dimensional coordinate systems, in accordance with some embodiments. The coordinate systems and sign conventions ofFIGS. 2 and 3are but mere examples, and other coordinate systems and sign conventions can also be used. Because well-defined equations exist to convert from any suitable three-dimensional coordinate system to any other suitable three-dimensional coordinate system, the three-dimensional positions can be saved, retrieved, and/or used by the processor in any suitable three-dimensional coordinate system.

In the graphical user interface discussed in detail below, a user is positioned above a horizontal plane, looking down on the horizontal plane. The origin, at position (0, 0, 0), is set to be the position of an observer, such as a listener. The x-axis can extend in a positive direction to the right of the origin. In some examples, the x-axis can correspond to a front-back direction. The z-axis can extend in a positive direction below the origin (e.g., downward when viewed on the graphical user interface). In some examples, the z-axis can correspond to a lateral direction. The y-axis can extend in a positive toward the viewer, so that y-values are positive “out of the plane”, and y-values are negative “into the plane”. In some examples, the y-axis can correspond to a height direction or an elevation direction, where positive y-values can correspond to heights above the listener, and negative y-values can correspond to heights below the listener. These x, y, and z directions can define a Cartesian coordinate system, so that a position200can be defined by three values (X, Y, Z).

For the graphical user interface discussed below, it is more convenient to discuss operations in the context of a spherical coordinate system. Many sign conventions and naming conventions exist for spherical coordinates. It should be noted that any of these sign conventions and naming conventions can be used with the graphical user interface discussed below, because one can easily be converted to another in a deterministic manner by known, well-defined equations.

The first of three spherical coordinate quantities is an azimuthal angle, denoted by angle φ, and expressed in degrees. Alternatively, radians or any other suitable angular units can also be used. The values of the azimuthal angle extend from zero degrees, inclusive, to 360 degrees, exclusive. In the context of the graphical user interface shown below, the azimuthal angle is zero along the negative z-axis (upward, from the origin), and increases in a clockwise direction (to the right, from the negative z-axis). Along the positive x-axis (to the right, from the origin), the azimuthal angle is 90 degrees. Along the positive z-axis (downward, from the origin), the azimuthal angle is 180 degrees. Along the negative x-axis (to the left, from the origin), the azimuthal angle is 270 degrees.

The second of three spherical coordinate quantities is an elevation angle, denoted by angle θ, and expressed in degrees. Alternatively, radians or any other suitable angular units can also be used. The values of the elevation angle extend from −90 degrees, inclusive, to +90 degrees, inclusive. In the context of the graphical user interface shown below, the elevation angle is zero at a height of zero, in the plane where y=0. Out of the page (e.g., toward the viewer), the elevation angle increases to a maximum value of +90 degrees, at which the three-dimensional position is located at the top of a virtual sphere. Into the page (e.g., away from the viewer), the elevation angle decreases to a minimum value of −90 degrees, at which the three-dimensional position is located at the bottom of the virtual sphere. In practice, in order to avoid losing azimuthal information, the graphical user interface can prevent the elevation angle from truly equaling −90 degrees or +90 degrees, instead confining its values between −89.9 degrees and +89.9 degrees, or between other suitable values that are close to but not equal to −90 degrees and +90 degrees, respectively. In this context, the elevation angle can approach −90 degrees or +90 degrees.

The third of three spherical coordinate quantities is a radius, denoted by quantity R, and expressed in units of distance or units of normalized distance. The values of radius extend from zero, inclusive, to any suitable positive value.

It is straightforward to convert between the spherical coordinates discussed above and Cartesian coordinates.FIG. 3shows the specific relationships between these coordinate systems. For Cartesian coordinates (X, Y, Z), quantity X equals R cos θ sin φ, quantity Y equals R sin θ, and quantity Z equals −R cos θ cos φ. These equations can easily be inverted to convert from Cartesian coordinates to spherical coordinates.

It should be noted that the spherical coordinate system provides a convenient way of describing the various positional quantities but does not require that calculations be explicitly performed in spherical coordinates, or that positions be saved in spherical coordinate. In practice, many or all of the underlying calculations involved with determining the three-dimensional position are carried out in Cartesian coordinates. For example, when we say that positioning a puck on a graphical user interface “determines” a radius, we mean that the graphical user interface can read x- and y-location values of the puck location from the graphical user interface and can calculate corresponding x- and y-components of the three-dimensional position, such that the x- and y-components can be described by a radius value. In other words, “determining” a radius does not explicitly require that the processor carries out calculations to explicitly compute the radius value directly. Instead, the process can carry out calculations in any suitable manner to compute the three-dimensional position, and the three-dimensional position can have a radius value. Similar discussion applies to determining an azimuthal angle and determining an elevation angle.

FIG. 4shows an example of a graphical user interface400for specifying a three-dimensional position, in accordance with some embodiments. The graphical user interface400can be used with system100, or with other suitable systems, to specify a three-dimensional position. The graphical user interface400ofFIG. 4is but one example of a graphical user interface; other suitable graphical user interfaces can also be used.

The graphical user interface400can include a puck402that is user-positionable in two dimensions to specify the three-dimensional position404, such that a distance between the puck402and a reference point406on the graphical user interface400can determine a radius of the three-dimensional position404, and an azimuthal orientation of the puck402with respect to the reference point406can determine an azimuthal angle of the three-dimensional position404. The graphical user interface400can further include an elevation controller408, separate from the puck402, that is user-adjustable to specify an elevation of the three-dimensional position404.

As explained above, using spherical coordinates merely provides a convenient way to describe the positions, and does not require that any underlying calculations are performed in spherical coordinates. In the example ofFIG. 4, the azimuthal angle φ is 27.0 degrees, the elevation angle θ is 45.0 degrees, the radius R is 1.30 normalized units, Cartesian coordinate X equals R cos θ sin φ, or 0.42 normalized units, Cartesian coordinate Y equals R sin θ, or 0.92 normalized units, and Cartesian coordinate Z equals −R cos θ cos φ, or −0.82 normalized units.

The puck402can be shown as any suitable graphic element or icon, such as a circle, a filled circle, or others. The puck402can be user-positionable in two dimensions to specify the three-dimensional position404. In the example ofFIG. 4, the puck can be dragged by a cursor410, although any suitable positioning mechanism can be used.

As explained above, a distance between the puck402and a reference point406on the graphical user interface400can determine a radius of the three-dimensional position404, and an azimuthal orientation of the puck402with respect to the reference point406can determine an azimuthal angle of the three-dimensional position404. To visually assist in positioning the puck402, the graphical user interface400can optionally include one or more indicator lines.

A zero-degree indicator line412can extend vertically through the reference point406and can indicate a direction of zero azimuth angle. A 45-degree indicator line414can extend through the reference point406and can indicate a direction of a 45-degree azimuth angle. A 90-degree indicator line416can extend horizontally through the reference point406and can indicate a direction of a 90-degree azimuth angle. A 135-degree indicator line418can extend through the reference point406and can indicate a direction of a 135-degree azimuth angle. In some examples, the indicator lines412,414,416, and418remain motionless on the graphical user interface400as the puck402is repositioned. A radial indicator line420can extend concentrically around the reference point406. In some examples, the radial indicator420can take the form of shading and/or a color gradient within the graphical user interface400, rather than a line.

A height indicator422can provide a visual indication of the elevation of the three-dimensional position404. The height indicator422may not be user-positionable. The height indicator422can include an icon or graphic element that optionally differs from the puck402, such as an open circle or others. The height indicator422can be positioned on a virtual line424extending between the puck402and the reference point406. A distance between the height indicator422and the reference point406on the graphical user interface400can correspond to a radius of the three-dimensional position404multiplied by a cosine of an elevation angle of the three-dimensional position404, such that the height indicator422is coincident with the puck402when the elevation angle is zero and approaches the reference point406when the elevation angle approaches −90 degrees or +90 degrees.

In some examples, the height indicator422and/or the puck402(and/or other indicators such as the virtual line424) can be colored to provide a further indication of height. For example, one or both of these elements can have a first color when the height is positive, and a second color different from the first color, when the height is negative. In some examples, the color can change abruptly when going from positive to negative or negative to positive. In other examples, the color can change gradually.

The graphical user interface400can optionally include a zoom controller, which can radially expand or contract all the elements shown with the puck402, so that the puck402and height indicator422, together, move toward or away from the reference point406and thus trade off the effective radial control range with radial accuracy within the space of the GUI elements. The zoom controller elements446,448, and450can optionally be labeled with a zoom controller label452. The zoom controller can include a zoom slider446and a zoom slider controller448that is user-positionable along the zoom slider446to set a value of the display zoom. The zoom controller can further include a zoom direct entry window450that can allow direct entry of a numerical value of the zoom. In some examples, when a user directly enters a numerical value into the zoom direct entry window450, the zoom slider controller448automatically moves to the corresponding location on the zoom slider446. In some examples, when a user positions the zoom slider controller448along the zoom slider446, the zoom direct entry window450automatically displays the numerical value corresponding to the zoom slider controller448position.

The elevation controller408can optionally include two separate controllers that adjust the elevation in different ways. Other configurations may provide only one control, but facilitate mode selection as to which operational mode is active for a given object position.

A first controller, referred to as an elevation angle controller, can specify an elevation angle of the three-dimensional position. When the first controller is adjusted, the radius and the azimuthal angle remain unchanged, and the three-dimensional position moves along the surface of a sphere.

A second controller, referred to as an elevation height controller, can specify an elevation height value of the three-dimensional position, the elevation height value being a product of the radius of the three-dimensional position and a sine of an elevation angle of the three-dimensional position. (The elevation height is essentially the Cartesian y-coordinate of the three-dimensional position404.) When the second controller is adjusted, the three-dimensional position moves vertically upward or downward, so that the azimuthal angle remains unchanged, but the radius grows or shrinks to keep the three-dimensional position on the surface of the (growing or shrinking) sphere.

These two controllers are discussed in detail below.

The elevation angle controller elements426,428,430,432, and434can optionally be labeled with an elevation angle controller label436. In some examples, the elevation angle can be referred to as a pitch.

The elevation angle controller can include an elevation angle slider426and an elevation angle slider controller428that is user-positionable along the elevation angle slider426, such that the following three effects occur. First, when the elevation angle slider controller428approaches a first end of the elevation angle slider, the elevation angle approaches −90 degrees and the three-dimensional position404approaches a bottom of a virtual sphere having the determined radius. Second, when the elevation angle slider controller428approaches a second end of the elevation angle slider426, opposite the first end, the elevation angle approaches +90 degrees and the three-dimensional position approaches a top of the virtual sphere. Third, when the elevation angle slider controller428moves between the first and second ends of the elevation angle slider426, the three-dimensional position moves along a surface of the virtual sphere.

The elevation angle controller can further include an elevation angle direct entry window430that can allow direct entry of a numerical value of the elevation angle. In some examples, when a user directly enters a numerical value into the elevation angle direct entry window430, the elevation angle slider controller428automatically moves to the corresponding location on the elevation angle slider426. In some examples, when a user positions the elevation angle slider controller428along the elevation angle slider426, the elevation angle direct entry window430automatically displays the numerical value corresponding to the elevation angle slider controller428position.

The elevation angle controller can optionally further include a first elevation angle preset button432, the pressing of which sets the elevation angle to a first elevation angle preset value. In the example ofFIG. 4, the first elevation angle preset value is +45 degrees, although other suitable values can also be used. The elevation angle controller can optionally further include a second elevation angle preset button434, the pressing of which sets the elevation angle to a second elevation angle preset value, different from the first elevation angle preset value. In the example ofFIG. 4, the second elevation angle preset value is 0 degrees, although other suitable values can also be used. More or fewer than two elevation angle preset buttons can also be used.

The elevation height controller elements438,440, and442can optionally be labeled with an elevation height controller label444. In some examples, the elevation height can be referred to as simply “y” or other suitable Cartesian coordinate axis designation.

The elevation height controller can include an elevation height slider438and an elevation height slider controller440that is user-positionable along the elevation height slider438, such that when the elevation height slider controller440moves from a first position on the elevation height slider438corresponding to a first positive elevation height, to a second position on the elevation height slider438corresponding to a second positive elevation height greater than the first positive elevation height, the following three effects occur. First, the three-dimensional position404moves along a line that is orthogonal to a plane defined by a locus of positions at which the elevation angle is zero and moves farther away from the plane. In the coordinate system ofFIG. 2, the three-dimensional position404moves along a line that is parallel to the y-axis. Second, the radius of the three-dimensional position404increases from a first radius value to a second radius value. Third, the elevation angle of the three-dimensional position404increases from a first positive elevation angle value to a second positive elevation angle value.

The elevation height controller can further include an elevation height direct entry window442that can allow direct entry of a numerical value of the elevation height (e.g., the actual Cartesian y-coordinate). In some examples, when a user directly enters a numerical value into the elevation height direct entry window442, the elevation height slider controller440automatically moves to the corresponding location on the elevation height slider438. In some examples, when a user positions the elevation height slider controller440along the elevation height slider438, the elevation height direct entry window442automatically displays the numerical value corresponding to the elevation height slider controller440position.

The elevation height controller can optionally further include one or more elevation height preset buttons (not shown inFIG. 4), the pressing of which sets the elevation height to a specified value.

Because the elevation controller408can include two separate controllers that adjust the elevation in different ways, the following two conditions can occur. First, adjusting an elevation angle with the elevation angle controller changes an elevation height value shown on the elevation height controller and does not move the puck402on the graphical user interface400. Second, adjusting an elevation height with the elevation height controller changes an elevation angle value shown on the elevation angle controller and moves the puck402radially with respect to the reference point406on the graphical user interface400.

Thus far, it has been assumed that the motion of the puck402and the elevation controller408, together, can freely determine the three-dimensional position404without any constraints on the three-dimensional position404. As a result, this mode of operation can be referred to a “free panner” mode.

There may be instances when constraints are desirable, because they can help move from one three-dimensional position to another. For example, an airplane flying overhead may pass from one three-dimensional position to another, at the same elevation height (e.g., the same Cartesian y-value).

To accommodate these instances, the graphical user interface400can include a mode selector454that allows locking and unlocking of a position parameter, such that when the position parameter is locked, user-initiated movement of the puck402can allow specification of the three-dimensional position404only along a surface of a virtual plane defined by the locked position parameter or over a surface of a virtual sphere defined by the locked position parameter.

In some examples, the mode selector454can receive a user selection of a radius lock mode, such that while the radius lock mode is selected, the radius of the three-dimensional position404is fixed at a fixed radius value, and user-initiated movement of the puck402allows specification of the three-dimensional position only over a surface of a virtual sphere having the fixed radius value.

In some examples, the mode selector454can receive a user selection of a lateral lock mode, such that while the lateral lock mode is selected, user-initiated movement of the puck402allows specification of the three-dimensional position404only within a virtual plane (e.g., a plane parallel to the y-z plane) that is orthogonal to a lateral direction (e.g., the x-direction, along416).

In some examples, the mode selector454can receive a user selection of a front/back lock mode, such that while the front/back lock mode is selected, user-initiated movement of the puck402allows specification of the three-dimensional position404only within a virtual plane (e.g., a plane parallel to the x-y plane) that is orthogonal to a front/back direction (e.g., the z-direction, along412), the front/back direction (e.g., the z-direction, along412) being orthogonal to the lateral direction (e.g., the x-direction, along416).

In some examples, the mode selector454can receive a user selection of a height lock mode, such that while the height lock mode is selected, user-initiated movement of the puck402allows specification of the three-dimensional position404only within a virtual plane (e.g., a plane parallel to the x-z plane) having a constant elevation. In this mode, user-initiated movement of the puck402in the plane of the graphical user interface400moves the three-dimensional position404in the x- and z-directions, with a fixed value for y. The result of such movements may be indicated as a change in elevation angle in the displayed values of the three-dimensional position404.

In some examples, the graphical user interface400can optionally include a first radius preset button456, the pressing of which sets the radius to a first radius preset value. In the example ofFIG. 4, the first radius preset value is zero, although other suitable values can also be used. In some examples, the first radius preset value can be slightly larger than zero, so that azimuth and elevation information are not lost when the radius is set to the first radius preset value. The graphical user interface400can optionally further include a second radius preset button458, the pressing of which sets the radius to a second radius preset value, different from the first radius preset value. In the example ofFIG. 4, the second radius preset value is 0.25, although other suitable values can also be used. The graphical user interface400can optionally further include a third radius preset button460, the pressing of which sets the radius to a third radius preset value, different from the first and second radius preset values. In the example ofFIG. 4, the third radius preset value is 1.0, although other suitable values can also be used. More or fewer than three radius preset buttons can also be used.

Thus far, there has been discussion of how to specify the three-dimensional position404. At this point, it is beneficial to discuss what would be physically placed at the determined position.

For examples in which the system performs audio mixing of sound by placing a virtual sound source at the determined three-dimensional position404, it can sometimes be desirable to impart a volume over which the virtual sound spreads. For example, a swarm of bees can produce a sound that appears to emanate from a full volume of the swarm, rather than from a single point.

To accommodate instances in which the virtual object to be placed at the determined three-dimensional position404has a specified volume, the graphical user interface400can include a spread controller, separate from the puck402, that is user-adjustable to specify a volumetric spread of the simulated audio signal.

The spread controller elements462,464,466,468, and470can optionally be labeled with a spread controller label472.

The spread controller can include a spread slider462and a spread slider controller464that is user-positionable along the spread slider462to set a value of the volumetric spread of the virtual object (such as a simulated audio signal). When the spread slider controller464is positioned at a first end of the spread slider462, the volumetric spread is zero. As the spread slider controller464is positioned away from the first end of the spread slider462, the volumetric spread of the virtual object increases.

The spread controller can further include a spread direct entry window466that can allow direct entry of a numerical value of the volumetric spread. In some examples, when a user directly enters a numerical value into the spread direct entry window466, the spread slider controller464automatically moves to the corresponding location on the spread slider462. In some examples, when a user positions the spread slider controller464along the spread slider462, the spread direct entry window466automatically displays the numerical value corresponding to the spread slider controller464position.

The spread controller can further include a spread mode selector470, labeled by spread mode selector label468. The spread mode selector470can receive a user selection of a spread mode as being exactly one of: a source-relative spread mode, in which the volumes of virtual objects are spread over a volume proximate the specified three-dimensional position, or an observer/listener-relative spread mode, in which the volumes of virtual objects are spread over a volume proximate a three-dimensional position corresponding to the reference point on the graphical user interface.

In some examples, the volume can be spread over any suitable shape, with suitable symmetry or with controllable/defined orientation, including a sphere, a directional cone, and others. In the specific example ofFIG. 4, the spread is over a sphere of a specified radius, centered at the origin or at the determined three-dimensional position404.

Thus far, it has been assumed that a single virtual object is to be positioned at the determined three-dimensional position404. There may be instances when it may be desirable to position multiple virtual objects at or near the three-dimensional position404. For examples in which the system performs audio mixing of sound, it can be desirable to locate two sound sources together with a specified relationship between them. For example, a twin-engine airplane can produce sound independently from its two engines. The two engines can have a fixed relationship to one another, such as a fixed spacing. During normal operation, the engines remain at the same height as each other, with a spacing in the horizontal direction. During a fancy aerobatic maneuver, such as a roll, the engines can move helically about one another, while maintaining the same spacing between the engines.

To accommodate these multiple related virtual objects, the graphical user interface can include a source mode selector474, which can specify how many virtual objects are to be placed and specify the positional relationship between the virtual objects.

In some examples, including the specific configuration shown inFIG. 4, the source mode selector474can receive a user selection of a single virtual object. For examples in which the system performs audio mixing of sound, this selection can correspond to a monaural audio signal. For this selected mode, the graphical user interface resembles the graphical user interface400ofFIG. 4, where the user has selected a source mode of a mono source, and the height indicator422is a single graphic element positioned on the virtual line424.

In some examples, the source mode selector474can receive a user selection of multiple virtual objects.FIGS. 5 and 6show examples of graphical user interfaces that can accommodate multiple virtual objects.

FIG. 5shows an example of a graphical user interface500in which a user has selected a stereo orbit mode on a source mode selector574, in accordance with some embodiments. The graphical user interface500can be used with system100, or with other suitable systems, to specify a three-dimensional position simultaneously for multiple virtual objects, such as audio sources. The graphical user interface500ofFIG. 5is but one example of a graphical user interface; other suitable graphical user interfaces can also be used. Elements numbered502-572inFIG. 5are similar in structure and function to corresponding elements numbered402-472fromFIG. 4.

Compared with the graphical user interface400ofFIG. 4, in which a user has selected a mono audio source, the graphical user interface500ofFIG. 5accommodates a stereo audio source. The puck502and elevation controller508can position both audio sources simultaneously. To show the positions of both audio sources, the graphical user interface500has replaced the single height indicator422ofFIG. 4with a left channel height indicator576and a right channel height indicator578.

In the example ofFIG. 5, the left and right channel height indicators576,578are at the same height and radius, and are azimuthally angled on opposite sides of the on a virtual line524extending between the puck502and the reference point506. In some examples, left and right channel height indicators576,578can have a virtual center that is positioned on the virtual line524. In other examples, the left channel height indicator576can vary in height and/or radius from the right channel height indicator578.

In the “orbit” mode selected at574, user-initiated movement of the puck502can move both the left channel height indicator576and the right channel height indicator578, together, in a motion that revolves around the reference point506. For example, if the puck502is moved clockwise in a circle that surrounds the reference point506, both the left channel height indicator576and the right channel height indicator578also move in respective circles around the reference point506, but do so maintaining their symmetric relationship with the virtual line524, where the left channel height indicator576is at an azimuth angle less than that of the virtual line524, and the right channel height indicator578is at an azimuth angle greater than that of the virtual line524.

FIG. 6shows an example of a graphical user interface600in which a user has selected a stereo standard mode on a source mode selector674, in accordance with some embodiments. The graphical user interface600can be used with system100, or with other suitable systems, to specify a three-dimensional position simultaneously for multiple virtual objects, such as audio sources. The graphical user interface600ofFIG. 6is but one example of a graphical user interface; other suitable graphical user interfaces can also be used. Elements numbered602-678inFIG. 5are similar in structure and function to corresponding elements numbered502-578fromFIG. 4.

Compared with the orbit mode ofFIG. 5, the left channel height indicator676and the right channel height indicator678now have a fixed spatial relationship between them, even as they are simultaneously repositioned by the puck602. For example, the right channel height indicator678remains directly to the right of the left channel height indicator676, regardless of where the puck602is moved. As a specific example, the standard mode is suitable for a two-engine passenger aircraft.

In the configurations ofFIGS. 5 and 6, there can be additional controls (not shown) that can specify the spatial relationship between the left and right virtual objects.

In some examples, there can be additional selectable modes that can accommodate more than two virtual objects, such as n objects, or a user-specified number of objects. Such modes can be suitable for relatively large numbers of objects that maintain a generally fixed relationship to one another, such as a flock of seagulls flying overhead in formation.

In other examples, there can be additional selectable modes that can accommodate scene representations which operate as a unit. An audio-related example would be one of an ambisonic soundfield representation. Such a representation has one or more audio signals positioned in a single mix. In such a mode, the three-dimensional position controlled by the graphical interface is the orientation of the soundfield, where changes in azimuth or elevation angle equate to rotation of the soundfield. Similarly, positional information might equate to translations and radial changes might change the perceptual depth of the soundfield when rendered. It is understood that the specific mappings are dependent on the rendering capabilities of the audio representation and this is but one example.

FIG. 7shows a flowchart of an example of a method700for specifying a three-dimensional position with a two-dimensional graphical user interface, in accordance with some embodiments. The method700can be executed using a processor coupled to any suitable graphical user interface, such as400,500, or600.

At operation702, a user-specified three-dimensional position can be received through a graphical user interface coupled to a processor and displayed on a display, the graphical user interface including a puck that is user-positionable in two dimensions to specify the three-dimensional position, such that a distance between the puck and a reference point on the graphical user interface determines a radius of the three-dimensional position, and an azimuthal orientation of the puck with respect to the reference point determines an azimuthal angle of the three-dimensional position, the graphical user interface further including an elevation controller, separate from the puck, that is user-adjustable to specify an elevation of the three-dimensional position.

At operation704, data representing the specified three-dimensional position can be stored on a storage device coupled to the processor.

The method700can optionally further include receiving, with the processor, an audio signal; and generating, with the processor, a mixed audio signal that corresponds to a simulation of the audio signal propagating from the specified three-dimensional position.

Many other variations than those described herein will be apparent from this document. For example, depending on the embodiment, certain acts, events, or functions of any of the methods and algorithms described herein can be performed in a different sequence, can be added, merged, or left out altogether (such that not all described acts or events are necessary for the practice of the methods and algorithms). Moreover, in certain embodiments, acts or events can be performed concurrently, such as through multi-threaded processing, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than sequentially. In addition, different tasks or processes can be performed by different machines and computing systems that can function together.

The various illustrative logical blocks, modules, methods, and algorithm processes and sequences described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, and process actions have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. The described functionality can be implemented in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of this document.

Embodiments of the automatic system and method described herein are operational within numerous types of general purpose or special purpose computing system environments or configurations. In general, a computing environment can include any type of computer system, including, but not limited to, a computer system based on one or more microprocessors, a mainframe computer, a digital signal processor, a portable computing device, a personal organizer, a device controller, a computational engine within an appliance, a mobile phone, a desktop computer, a mobile computer, a tablet computer, a smartphone, and appliances with an embedded computer, to name a few.

Such computing devices can typically be found in devices having at least some minimum computational capability, including, but not limited to, personal computers, server computers, hand-held computing devices, laptop or mobile computers, communications devices such as cell phones and PDA's, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, audio or video media players, and so forth. In some embodiments the computing devices will include one or more processors. Each processor may be a specialized microprocessor, such as a digital signal processor (DSP), a very long instruction word (VLIW), or other micro-controller, or can be conventional central processing units (CPUs) having one or more processing cores, including specialized graphics processing unit (GPU)-based cores in a multi-core CPU.

The process actions or operations of a method, process, or algorithm described in connection with the embodiments of the automatic system and method disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in any combination of the two. The software module can be contained in computer-readable media that can be accessed by a computing device. The computer-readable media includes both volatile and nonvolatile media that is either removable, non-removable, or some combination thereof. The computer-readable media is used to store information such as computer-readable or computer-executable instructions, data structures, program modules, or other data. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media.

Computer storage media includes, but is not limited to, computer or machine readable media or storage devices such as Bluray discs (BD), digital versatile discs (DVDs), compact discs (CDs), floppy disks, tape drives, hard drives, optical drives, solid state memory devices, RAM memory, ROM memory, EPROM memory, EEPROM memory, flash memory or other memory technology, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other device which can be used to store the desired information and which can be accessed by one or more computing devices.

A software module can reside in the RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of non-transitory computer-readable storage medium, media, or physical computer storage known in the art. An exemplary storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an application specific integrated circuit (ASIC). The ASIC can reside in a user terminal. Alternatively, the processor and the storage medium can reside as discrete components in a user terminal.

The phrase “non-transitory” as used in this document means “enduring or long-lived”. The phrase “non-transitory computer-readable media” includes any and all computer-readable media, with the sole exception of a transitory, propagating signal. This includes, by way of example and not limitation, non-transitory computer-readable media such as register memory, processor cache and random-access memory (RAM).

The phrase “audio signal” is a signal that is representative of a physical sound.

Retention of information such as computer-readable or computer-executable instructions, data structures, program modules, and so forth, can also be accomplished by using a variety of the communication media to encode one or more modulated data signals, electromagnetic waves (such as carrier waves), or other transport mechanisms or communications protocols, and includes any wired or wireless information delivery mechanism. In general, these communication media refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information or instructions in the signal. For example, communication media includes wired media such as a wired network or direct-wired connection carrying one or more modulated data signals, and wireless media such as acoustic, radio frequency (RF), infrared, laser, and other wireless media for transmitting, receiving, or both, one or more modulated data signals or electromagnetic waves. Combinations of the any of the above should also be included within the scope of communication media.

Further, one or any combination of software, programs, computer program products that embody some or all of the various embodiments of the automatic system and method described herein, or portions thereof, may be stored, received, transmitted, or read from any desired combination of computer or machine-readable media or storage devices and communication media in the form of computer executable instructions or other data structures.

Embodiments of the automatic system and method described herein may be further described in the general context of computer-executable instructions, such as program modules, being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, and so forth, which perform particular tasks or implement particular abstract data types. The embodiments described herein may also be practiced in distributed computing environments where tasks are performed by one or more remote processing devices, or within a cloud of one or more devices, that are linked through one or more communications networks. In a distributed computing environment, program modules may be located in both local and remote computer storage media including media storage devices. Still further, the aforementioned instructions may be implemented, in part or in whole, as hardware logic circuits, which may or may not include a processor.

EXAMPLES

To further illustrate the device and related method disclosed herein, a non-limiting list of examples is provided below. Each of the following non-limiting examples can stand on its own or can be combined in any permutation or combination with any one or more of the other examples.

In Example 1, a system can include: a processor configured to: display a graphical user interface, receive a user-specified three-dimensional position through the graphical user interface, and store data representing the specified three-dimensional position on a storage device coupled to the processor, the graphical user interface comprising: a puck that is user-positionable in two dimensions to specify the three-dimensional position, such that a distance between the puck and a reference point on the graphical user interface determines a radius of the three-dimensional position, and an azimuthal orientation of the puck with respect to the reference point determines an azimuthal angle of the three-dimensional position; and an elevation controller, separate from the puck, that is user-adjustable to specify an elevation of the three-dimensional position.

In Example 2, the system of Example 1 can optionally be configured such that the elevation controller comprises an elevation angle controller configured to specify an elevation angle of the three-dimensional position.

In Example 3, the system of any one of Examples 1-2 can optionally be configured such that the elevation angle controller comprises an elevation angle slider and an elevation angle slider controller that is user-positionable along the elevation angle slider, such that: when the elevation angle slider controller approaches a first end of the elevation angle slider, the elevation angle approaches −90 degrees and the three-dimensional position approaches a bottom of a virtual sphere having the determined radius, when the elevation angle slider controller approaches a second end of the elevation angle slider, opposite the first end, the elevation angle approaches +90 degrees and the three-dimensional position approaches a top of the virtual sphere, and when the elevation angle slider controller moves between the first and second ends of the elevation angle slider, the three-dimensional position moves along a surface of the virtual sphere.

In Example 4, the system of any one of Examples 1-3 can optionally be configured such that the elevation angle controller further comprises: an elevation angle preset button, the pressing of which sets the elevation angle to an elevation angle preset value.

In Example 5, the system of any one of Examples 1-4 can optionally be configured such that the elevation controller comprises an elevation height controller configured to specify an elevation height value of the three-dimensional position, the elevation height value being a product of the radius of the three-dimensional position and a sine of an elevation angle of the three-dimensional position.

In Example 6, the system of any one of Examples 1-5 can optionally be configured such that the elevation height controller comprises an elevation height slider and an elevation height slider controller that is user-positionable along the elevation height slider, such that when the elevation height slider controller moves from a first position on the elevation height slider corresponding to a first positive elevation height, to a second position on the elevation height slider corresponding to a second positive elevation height greater than the first positive elevation height: the three-dimensional position moves along a line that is orthogonal to a plane defined by a locus of positions at which the elevation angle is zero and moves farther away from the plane, the radius of the three-dimensional position increases from a first radius value to a second radius value, and the elevation angle of the three-dimensional position increases from a first positive elevation angle value to a second positive elevation angle value.

In Example 7, the system of any one of Examples 1-6 can optionally be configured such that the elevation controller comprises an elevation angle controller configured to specify an elevation angle of the three-dimensional position, and an elevation height controller configured to specify an elevation height value of the three-dimensional position, the elevation height value being a product of the radius of the three-dimensional position and a sine of an elevation angle of the three-dimensional position, such that: adjusting an elevation angle with the elevation angle controller changes an elevation height value shown on the elevation height controller and does not move the puck on the graphical user interface; and adjusting an elevation height with the elevation height controller changes an elevation angle value shown on the elevation angle controller and moves the puck radially with respect to the reference point on the graphical user interface.

In Example 8, the system of any one of Examples 1-7 can optionally be configured such that the graphical user interface further comprises a mode selector that allows locking and unlocking of a position parameter, such that when the position parameter is locked, user-initiated movement of the puck allows specification of the three-dimensional position only along a surface of a virtual plane defined by the locked position parameter or over a surface of a virtual sphere defined by the locked position parameter.

In Example 9, the system of any one of Examples 1-8 can optionally be configured such that the mode selector is configured to receive a user selection of a radius lock mode, such that while the radius lock mode is selected, the radius of the three-dimensional position is fixed at a fixed radius value, and user-initiated movement of the puck allows specification of the three-dimensional position only over a surface of a virtual sphere having the fixed radius value.

In Example 10, the system of any one of Examples 1-9 can optionally be configured such that: the mode selector is configured to receive a user selection of a lateral lock mode, such that while the lateral lock mode is selected, user-initiated movement of the puck allows specification of the three-dimensional position only within a virtual plane that is orthogonal to a lateral direction; the mode selector is configured to receive a user selection of a front/back lock mode, such that while the front/back lock mode is selected, user-initiated movement of the puck allows specification of the three-dimensional position only within a virtual plane that is orthogonal to a front/back direction, the front/back direction being orthogonal to the lateral direction; the mode selector is configured to receive a user selection of a height lock mode, such that while the height lock mode is selected, user-initiated movement of the puck allows specification of the three-dimensional position only within a virtual plane having a constant elevation.

In Example 11, the system of any one of Examples 1-10 can optionally be configured such that the graphical user interface further comprises: a radius preset button, the pressing of which sets the radius of the three-dimensional position to a radius preset value.

In Example 12, the system of any one of Examples 1-11 can optionally be configured such that the graphical user interface further comprises: a height indicator configured to provide a visual indication of the elevation of the three-dimensional position, the height indicator being positioned on a virtual line extending between the puck and the reference point, a distance between the height indicator and the reference point on the graphical user interface corresponding to a radius of the three-dimensional position multiplied by a cosine of an elevation angle of the three-dimensional position, such that the height indicator is coincident with the puck when the elevation angle is zero and approaches the reference point when the elevation angle approaches −90 degrees or +90 degrees, the height indicator being exactly one of: a single graphic element positioned on the virtual line, the single graphic element corresponding to a position of a single virtual object, or a plurality of graphic elements, the plurality of graphic elements having a virtual center that is positioned on the virtual line, the plurality of graphic elements corresponding to positions of a plurality of virtual objects.

In Example 13, the system of any one of Examples 1-12 can optionally be configured such that the processor is further configured to receive an audio signal, and generate a mixed audio signal that corresponds to a simulation of the audio signal propagating from the specified three-dimensional position.

In Example 14, the system of any one of Examples 1-13 can optionally be configured such that the graphical user interface further comprises a spread controller, separate from the puck, that is user-adjustable to specify a volumetric spread of the simulated audio signal.

In Example 15, the system of any one of Examples 1-14 can optionally be configured such that the spread controller comprises: a spread slider and a spread slider controller that is user-positionable along the spread slider to set a value of the volumetric spread, such that when the spread slider controller is positioned at a first end of the spread slider, the volumetric spread of the simulated audio signal is zero.

In Example 16, the system of any one of Examples 1-15 can optionally further include a spread mode selector, the spread mode selector configured to receive a user selection of a spread mode as being exactly one of: a source-relative spread mode, in which the simulated audio signal is spread over a volume proximate the specified three-dimensional position, or a listener-relative spread mode, in which the simulated audio signal is spread over a volume proximate a three-dimensional position corresponding to the reference point on the graphical user interface.

In Example 17, a method can include: receiving a user-specified three-dimensional position through a graphical user interface coupled to a processor and displayed on a display, the graphical user interface including a puck that is user-positionable in two dimensions to specify the three-dimensional position, such that a distance between the puck and a reference point on the graphical user interface determines a radius of the three-dimensional position, and an azimuthal orientation of the puck with respect to the reference point determines an azimuthal angle of the three-dimensional position, the graphical user interface further including an elevation controller, separate from the puck, that is user-adjustable to specify an elevation of the three-dimensional position; and storing data representing the specified three-dimensional position on a storage device coupled to the processor.

In Example 18, the method of Example 17 can optionally further include: receiving, with the processor, an audio signal; and generating, with the processor, a mixed audio signal that corresponds to a simulation of the audio signal propagating from the specified three-dimensional position.

In Example 19, the method of any one of Examples 17-18 can optionally further include: spreading the simulated audio signal over a sphere having a specified radius.

In Example 20, a system can include: a processor configured to: display a graphical user interface, receive a user-specified three-dimensional position through the graphical user interface, and store data representing the specified three-dimensional position on a storage device coupled to the processor, the graphical user interface comprising: a puck that is user-positionable in two dimensions to specify the three-dimensional position, such that a distance between the puck and a reference point on the graphical user interface determines a radius of the three-dimensional position, and an azimuthal orientation of the puck with respect to the reference point determines an azimuthal angle of the three-dimensional position; an elevation controller, separate from the puck, that is user-adjustable to specify an elevation of the three-dimensional position; a mode selector that allows reversible locking of a position parameter, such that when the position parameter is locked, user-initiated movement of the puck allows specification of the three-dimensional position only along a surface of a virtual plane or over a surface of a virtual sphere; and a height indicator configured to provide a visual indication of the elevation of the three-dimensional position, the height indicator being positioned on a virtual line extending between the puck and the reference point, a distance between the height indicator and the reference point on the graphical user interface corresponding to a radius of the three-dimensional position multiplied by a cosine of an elevation angle of the three-dimensional position, such that the height indicator is coincident with the puck when the elevation angle is zero and approaches the reference point when the elevation angle approaches −90 degrees or +90 degrees, the height indicator being exactly one of: a single graphic element positioned on the virtual line, the single graphic element corresponding to a position of a single virtual object, or a plurality of graphic elements, the plurality of graphic elements having a virtual center that is positioned on the virtual line, the plurality of graphic elements corresponding to positions of a plurality of virtual objects.