Systems and methods for measuring image quality based on an image quality metric

The disclosed computer-implemented method may include encoding media content into a plurality of encoded media files, each encoded media file having an encoded resolution and at least one associated full reference metric, identifying one of the plurality of encoded media files to provide to a computing device based on at least a playback resolution of a display device included in the computing device, and weighting a measurement of a quality of the encoded media file based on the at least one associated full reference metric and on at least one characteristic associated with the display device. Various other methods, systems, and computer-readable media are also disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1is a diagram showing media content displayed on a display device of a mobile computing device.

FIG. 2is a diagram showing media content displayed on a monitor of a desktop computing device.

FIG. 3is a diagram showing media content displayed on a high-definition television (HDTV).

FIG. 4is a diagram showing an example first graph of spatial frequency verses contrast sensitivity for an example visual system for viewing media content.

FIG. 5is a diagram showing an example second graph of spatial frequency verses contrast sensitivity for an example visual system for viewing media content.

FIG. 6is a block diagram of an example system for providing media content from a content provider computing device to a computing system that may further provide an encoded version of the media content to one or more playback display devices included in respective playback computing devices.

FIG. 7is a block diagram of an example system for providing encoded content to a playback computing device.

FIG. 8is a diagram showing an example graph of predicted mean opinion scores based on an encoding mean opinion score verses a mean opinion score based on information provided by a user.

FIG. 9is a block diagram of an example system900that includes modules for use in measuring image quality based on an image quality metric.

FIG. 10illustrates an exemplary network environment in which aspects of the present disclosure may be implemented.

FIG. 11is a flow diagram of an exemplary computer-implemented method for measuring image quality based on an image quality metric.

FIG. 12is an illustration of an exemplary artificial-reality headband that may be used in connection with embodiments of this disclosure.

FIG. 13is an illustration of exemplary augmented-reality glasses that may be used in connection with embodiments of this disclosure.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

A user may view streaming media content on many different viewing devices that may include, but are not limited to, a smartphone, a tablet, a laptop computer, a desktop computer, and a high-definition television (HDTV). A user may experience a different viewing experience for the streaming media content dependent on the viewing device. A display device included in each viewing device may be of a different pixel resolution and a different pixel density. In addition, or in the alternative, a user may view the streaming media content on the display device at different distances. The quality of the display device and/or a viewing distance of the user to the display device may affect a perceived quality of the streaming media content as viewed by the user. Described herein are systems and methods for improving the measurement of perceived image quality based on image quality metrics for an encoded image file of an image or for an encoded file for streaming media content.

The present disclosure is generally directed to systems and methods for improvements in the measuring of a mean opinion score of a quality of streaming media content as perceived by a user. As will be explained in greater detail below, embodiments of the present disclosure may provide configurable multi-scale weighting for measuring a mean opinion score of a quality of streaming media content. For example, each streaming media content may be at a different original encoded resolution. In addition, each streaming media content may be viewed on various display devices of different pixel resolution and quality and may be viewed at various distances, in some cases, dependent on a size of the display device. A mean opinion score may be determined for each combination of display quality and viewing distance using configurable multi-scale weighting.

In some implementations, a full reference metric may be computed at each of one or more display device resolutions (pixel-width resolutions (measured in pixels per “x” direction (PPx)), e.g., 360 PPx, 720 PPx, 1080 PPx). A structural similarity index (SSIM) and/or a peak signal-to-noise ratio (PSNR) may be estimated for each display device resolution and stored with an encoding of the streaming content. At delivery time of the streaming media content to a viewing device of a user, a playback resolution for the streaming media content, a resolution of a display device included in the viewing device, and estimated viewing distance for viewing the streaming media content on the viewing device may be determined.

A measurement of a mean opinion score for the streaming media content may be weighted based on a spatial frequency which is a function of the display device resolution divided by the viewing distance of the user to the display device of the viewing device. The weighting may be applied at each scale of a multiscale mean opinion score weighting based on a contrast sensitivity of the display device at a corresponding spatial frequency. For example, a computing device (e.g., a mobile computing device) may include a type of 1080 PPx display device that has a high pixel density (e.g., a pixel density higher than another type of 1080 PPx device). Though two display devices may be of the same size and resolution, one may be of a higher pixel density (one may have a larger number of pixels per centimeter, a larger number of pixels per inch) than the other. In some cases, each of the pixels of the higher pixel density display may not be individually viewed at a typical viewing distance of a user to the computing device, creating a perceived impression of a sharper image as viewed by the user.

A first weighting may be applied to a mean opinion score for streaming media content viewed on the higher pixel density display device of the computing device based on the contrast sensitivity of the higher pixel density display device at a corresponding spatial frequency. Another factor that may impact the first weighting is a viewing distance between the user and the higher pixel density display device of the computing device. In another example, an HDTV may include a 1080 PPx display device. A second weighting may be applied to a mean opinion score for streaming media content viewed on the HDTV. The second weighting may be based on the contrast sensitivity of the HDTV display device at a corresponding spatial frequency and on a viewing distance between the user and the HDTV display device. The second weighting may be higher than the first weighting even though the resolution of each display device is 1080 PPx because the spatial frequency for the higher pixel density display device is higher than a spatial frequency of the HDTV display device and the sensitivity of the higher pixel density display device is lower as compared to a sensitivity of the HDTV display device.

A structural similarity (SSIM) index may provide an approximation of image quality as perceived by a user based on the assumption that the human visual system (e.g., the human eye) may be adapted for extracting structural information from a viewed image. Structural similarity for an image may be used for measuring a similarity between a reference image for the image (e.g., typically an image that is uncompressed and free of artifacts and/or distortion) and the image as displayed on a display device and viewed by a user.

A multi-scale structural similarity (MS-SSIM) index for an image may be determined over multiple scales by, for example, sub-sampling (down sampling) the image based on an original resolution of the image. A MS-SSIM index may be computed by computing a SSIM for each of multiple different image resolutions. A weighted MS-SIMM index may weigh the MS-SSIM index based on a spatial frequency sensitivity of the display device that the image is being viewed on.

FIG. 1is a diagram100showing media content102displayed on a display device104of a mobile computing device106. For example, the mobile computing device106may be a smartphone, personal digital assistant (PDA), tablet, or other type of mobile computing device. Media content may be viewed (displayed) in a portrait orientation or mode of operation (as shown inFIG. 1) or a landscape orientation or mode of operation.

A user (e.g., user108) may view the media content102on a screen of a display device104included in the mobile computing device106. The user108may view the media content102at a viewing distance112as measured from the eyes of the user108to the screen of the display device104. The media content102may be viewed at a viewing angle110. In some implementations, the viewing distance112may play a role in a user perceived image quality of the media content102as viewed by the user108on the screen of the display device104.

In some implementations, one or more characteristics of the display device104in combination with (in addition to) the viewing distance112may also impact the user perceived quality of the media content102. The one or more characteristics of a display device (e.g., the display device104) may include, but are not limited to, a quality of the display device (e.g., a pixel density, a pixel resolution), and a size of the display device (e.g., as measured by a display device diameter114).

The media content102delivered to the mobile computing device106for display on the display device104may be of a certain encoding (encoded resolution). As described herein, the encoding may be expressed in pixels per “x” direction (PPx). The encoding of the media content delivered to a display device (e.g., the display device104) may be of a number of pixels in the “x” direction that are equal to or slightly greater than (e.g., 1% greater, 5% greater, 10% greater) a number of pixels in the “x” direction of the screen of the display device (e.g., the display device104). In some implementations, the resolution of the encoding provided to the mobile computing device106for display on the display device104may be selected to maximize a number of pixels of the display device104while taking into account a size of the encoding file for the encoding, a transmission speed between the computing system providing the encoding file and the mobile computing device106, and a processing speed of the mobile computing device106when rendering the encoding file on the display device104. For example, for a given screen size (e.g., pixel resolution, number of pixels in the “x” direction) of a display device (e.g., the display device104), an encoding may be selected that is equal to or slightly larger than the screen size.

In some implementations, two display devices, a first display device and a second display device, may be of the same screen size (e.g., diagonals of each display device are equal), however, one of the display devices may be of a higher pixel density that the other. For example, a logical pixel on the first display device may be equal to a single physical pixel on the display device while a logical pixel on the second display device may be equal to more than one (e.g., four) physical pixels on the display device. For example, the second display device may be referred to as having a high pixel density (a pixel density higher than the first display device). The encoding may include image information and data for each logical pixel. However, the same encoding displayed on the second display device may be perceived by a user as having a higher perceived image quality based on the use of multiple physical pixels to represent each logical pixel in an encoding. In some implementations, however, this improved perceived image quality may be a factor of the viewing distance of the user to the display device. For example, if the viewing distance is too close to the display device, the user may see the individual physical pixels, which in some cases, may make an image appear as pixelated.

Referring toFIG. 1, the viewing distance112is greater than a size of the display device104(as represented by the display device diameter114(which may also be referred to as a diagonal114)). For example, in implementations where the display device104may be considered a high pixel density display device (e.g., multiple physical pixels may be used to represent each logical pixel in an encoding), it may be assumed that a viewing distance of a user to the display device104that is greater than a size of the display device104(e.g., a viewing distance112that is greater than the display device diameter114of the display device104) may result in a perceived image quality of the media content102that is better than (improved over, greater than) a perceived image quality of the media content102as displayed on a display device that represents each logical pixel as a physical pixel.

FIG. 2is a diagram200showing media content202displayed on a display device (e.g., monitor204) of a desktop computing device206. In some implementations, the desktop computing device206may be a laptop computing device, a notebook computing device, or other type of computing device that may include a monitor and a keyboard or other type of input device.

A user (e.g., user208) may view the media content202on a screen of the monitor204. The user208may view the media content202at a viewing distance212as measured from the eyes of the user208to the screen of the monitor204. The media content202may be viewed at a viewing angle210. In some implementations, the viewing distance212may play a role in a user perceived image quality of the media content202as viewed by the user208on the screen of the monitor204.

In some implementations, one or more characteristics of the monitor204in combination with (in addition to) the viewing distance212may also impact the user perceived quality of the media content202. The one or more characteristics of the monitor204may include, but are not limited to, a quality of the monitor (e.g., a pixel density, a pixel resolution), and a size of the display device (e.g., as measured by a display device diameter214).

The media content202delivered to the desktop computing device206for display on the monitor204may be of a certain encoding (encoded resolution). As described herein, the encoding may be expressed in pixels per “x” direction (PPx). The encoding of the media content delivered to the monitor204) may be of a number of pixels in the “x” direction that are equal to or slightly greater than (e.g., 1% greater, 5% greater, 10% greater) a number of pixels in the “x” direction of the screen of the monitor204. For example, for a given screen size (e.g., pixel resolution, number of pixels in the “x” direction) of a monitor (e.g., the monitor204), an encoding may be selected that is equal to or slightly larger than the screen size.

In some implementations, two monitors, a first monitor and a second monitor, may be of the same screen size (e.g., diagonals of each monitor are equal), however, one of the monitors may be of a higher pixel density that the other. As described with reference toFIG. 1, the same encoding displayed on each monitor may be perceived by a user as having a higher perceived image quality on the second monitor based on the use of multiple physical pixels to represent each logical pixel in an encoding. In some implementations, however, this improved perceived image quality may be a factor of the viewing distance of the user to the monitor. For example, if the viewing distance is too close to the monitor, the user may see the individual physical pixels, which in some cases, may make an image appear as pixelated.

Referring toFIG. 2, the viewing distance212may be approximately equal to or greater than a size of the monitor204(as represented by the display device diameter214(which also may be referred to as diagonal214)). For example, in implementations where the monitor204may be considered a high pixel density monitor (e.g., multiple physical pixels may be used to represent each logical pixel in an encoding), it may be assumed that the viewing distance212may result in a perceived image quality of the media content202that is better than (improved over, greater than) a perceived image quality of the media content202as displayed on a monitor that represents each logical pixel as a physical pixel.

FIG. 3is a diagram300showing media content302displayed on a high-definition television (HDTV)306. A user (e.g., user308) may view the media content302on a screen304of the HDTV306. The user308may view the media content302at a viewing distance312as measured from the eyes of the user308to the screen304. The media content302may be viewed at a viewing angle310. In some implementations, the viewing distance312may play a role in a user perceived image quality of the media content302as viewed by the user308on the screen304.

In some implementations, one or more characteristics of the HDTV306in combination with (in addition to) the viewing distance312may also impact the user perceived quality of the media content302. The one or more characteristics of the HDTV306may include, but are not limited to, a quality of the HDTV (e.g., a pixel density, a pixel resolution), and a size of the HDTV (e.g., as measured by display device diameter314).

The media content302delivered to the HDTV306for display on the screen304may be of a certain encoding (encoded resolution). As described herein, the encoding may be expressed in pixels per “x” direction (PPx). The encoding of the media content delivered to the HDTV306may be of a number of pixels in the “x” direction that are equal to or slightly greater than (e.g., 1% greater, 5% greater, 10% greater) a number of pixels in the “x” direction of the screen304. For example, for a given size of the screen304(e.g., pixel resolution, number of pixels in the “x” direction), an encoding may be selected that is equal to or slightly larger than the size of the screen304.

Referring toFIG. 3, the viewing distance112may impact a perceived image quality. For example, if the user308were to sit very close to the screen304(e.g., a viewing distance much less than the display device diameter314(e.g., a viewing distance equal to half the display device diameter314(which may also be referred to as diagonal314)), the user308may see each physical pixel on the screen304. Viewing the media content302at the viewing distance312, however, may result in a visual smoothing or blending of pixels as viewed by the user308, resulting in a better (higher) perceived image quality than at the much closer viewing distance.

FIG. 4is a diagram showing an example first graph400of spatial frequency402(in cycles per degree) verses contrast sensitivity404for an example visual system for viewing media content.

FIG. 5is a diagram showing an example second graph500of spatial frequency502(in cycles per degree) verses contrast sensitivity504for an example visual system for viewing media content.

A sensitivity of a visual system may be dependent on a spatial frequency of a signal. Human visual sensitivity may peak at frequencies around four cycles per degree of visual angle and may decrease along high and low frequency directions. Spatial frequency may be dependent on a number of pixels, a pixel density (pixel per inch (ppi), pixels per “x” direction (ppx)) and viewing distance of a user from a display device.

In some implementations, a mean opinion score may guide the customizing of media content based on a viewing distance of the user to the display device displaying (playing) the media content. For example, as described, display devices may have different pixel densities (pixels per centimeter, pixels per inch) that may be independent of a size of the display device. In some implementations, a perceived image quality may be dependent on pixel resolution, pixel density and a viewing distance of a user from the display device. For example, a user viewing 1080p encoded media content on a low ppi computing device may have a different experience as compared to viewing the 1080p encoded media content on a high ppi computing device.

In some implementations, it may be beneficial to have a mean opinion score configurable based on a multi-scale weighting to accommodate for display device number of pixels, pixel density (ppi), and viewing distance of a user from a display device. For example, a resolution of received media content may vary. In addition, or in the alternative, as shown with reference toFIGS. 1-3, the media content may be viewed on display devices of varying pixel densities, varying sizes, and may be viewed at different viewing distances. All of these factors may contribute to the viewing experience of the user.

FIG. 6is a block diagram of an example system600for providing media content632from a content provider computing device626to a computing system612that may further provide an encoded version of the media content to one or more playback display devices (e.g., display devices606a-e) included in respective playback computing devices (e.g., computing devices608a-e).

The media content632may be stored in a received content storage repository616. In some implementations, the media content632may be an encoded or transcoded version of original media content as captured, obtained, generated, and/or created by the content provider computing device626. A transcoder602may perform the encoding and/or transcoding on the original media content to compress the data in the original media content for improved transmission from the content provider computing device636to the computing system612. In some implementations, the received media content may be the original media content.

As described herein, resolution of media content may be represented as video resolution where “p” stands for progressive scanned (i.e., non-interlaced). For example, a number before the “p” may represent a number of vertical pixels in the media content. A number of horizontal pixels may be determined to provide media content at particular aspect ratios (e.g., 3:2, 4:3, 16:9, 5:3, 15:10, 18:10, etc.). Though examples are provided herein, the concepts described may be applied to media content at all resolutions.

For example, 240p may have a resolution of 320 pixels horizontally by 240 pixels vertically to provide media content at a 4:3 aspect ratio. For example, 240p may have a resolution of 428 pixels horizontally by 240 pixels vertically to provide media content at a 16:9 aspect ratio. For example, 360p may have a resolution of 480 pixels horizontally by 360 pixels vertically to provide media content at a 4:3 aspect ratio.

For example, 480p may have a resolution of 640 pixels horizontally by 480 pixels vertically to provide media content at a 4:3 aspect ratio. For example, 480p may have a resolution of 854 pixels horizontally by 480 pixels vertically to provide media content at a 16:9 aspect ratio. For example, 480p may have a resolution of 720 pixels horizontally by 480 pixels vertically to provide media content at a 3:2 aspect ratio.

For example, 720p may have a resolution of 1280 pixels horizontally by 720 pixels vertically to provide media content at a 16:9 aspect ratio. For example, 1080p may have a resolution of 1920 pixels horizontally by 1080 pixels vertically to provide media content at a 16:9 aspect ratio. For example, 4K may have a resolution of 3840 pixels horizontally by 2160 pixels vertically to provide media content at a 16:9 aspect ratio.

An encoding module630may encode received content stored in the received content storage repository616into one or more progressive scanned encodings (encodings604a-f) for storage in the encoded content storage638. An adaptive bitrate module634may interface with one or more of the playback computing devices608a-eto determine which progressive scanned encoding of the media content to provide to the respective playback display devices606a-ebased on one or more criteria that may include, but is not limited to, a display resolution of the respective playback display device606a-e, an orientation of the respective playback display device606a-e, a viewing mode of the media content, and a communication speed between the computing system612and the respective playback computing device608a-e.

For example, the computing device608amay be a smartphone, personal digital assistant (PDA), or other type of mobile computing device. Media content may be viewed (displayed) in a portrait orientation or mode of operation (e.g., portrait mode610) or a landscape orientation or mode of operation (e.g., landscape mode620). For example, the computing device608bmay be a tablet computing device. Media content may be viewed (displayed) in a portrait orientation or mode of operation (e.g., portrait mode622) or a landscape orientation or mode of operation (e.g., landscape mode624). For example, the computing device608cmay be a laptop computing device. For example, the computing device608dmay be a desktop computing device. For example, the computing device608emay be an HDTV.

As shown inFIG. 6, the same media content (e.g., a picture (image, video) of a hot air balloon) may be displayed on each playback display device606a-e. A quality of the received media content may influence a playback quality of the displayed encoded media content. For example, dependent on the size, pixel density, resolution, and orientation of the playback display device606a-e, as well as a viewing distance of the user to the playback device (as shown, for example, inFIGS. 1-3), a playback quality of the displayed encoded media content may vary and, also taking into account a viewing distance of the user to the playback display device, a quality of the displayed encoded media content as perceived by the user may vary. For example, the same media content may be displayed on the display device606aof the computing device608ain a portrait mode610and a full screen landscape mode620. In some implementations, the media content may be displayed on the display device606aof the computing device608ain an in-feed portrait mode. In some cases, a playback quality of the media content may be perceived as lower quality when displayed in the full screen landscape mode620as compared to the portrait mode610if the quality (resolution) of the encoded media content provided to the computing device608ain each orientation was received at a resolution such that the portrait mode610provides a sharper image than the full screen landscape mode620. For example, displaying encoded media content that was received at a resolution that is much lower (less than) a resolution of the display device606eof the computing device608e(e.g., a 4K HDTV) may result in a perceived low quality for the viewing of the media content on the display device606e.

FIG. 7is a block diagram of an example system700for providing encoded content to a playback computing device (e.g., playback computing device722). The playback computing device722may be one of the playback computing devices608a-e. The playback computing device722may include a display device724. The display device724may be one of the display devices606a-e.

In some implementations, referring also toFIG. 6, a reference metric module702included in the encoding module630may compute a full reference metric at each encoding resolution (e.g.,240pencoding604a,360p encoding604b,480p encoding604c,720p encoding604d,1080p encoding604e, and 4K encoding604e). For example, a scaled structural similarity (SSIM) and/or a peak signal-to-noise ratio (PSNR) may be estimated for the media content at a fixed resolution and stored as the full reference metric with the associated encoded media content stored in the encoded content storage638.

In some implementations, the reference metric module702may determine (compute) a multi-scale SSIM (MS-SSIM) index for an image (or a video) at an encoding at one or more down sampled scales. The MS-SIMM index may be considered a full reference metric that may be stored along with the encoding in the encoded content storage638. The use of a MS-SSIM index for an image may allow for the incorporation of image details at different resolutions (e.g., the one or more down sampled scales) for the image. For example, the original image may be indexed as Scale_1 and an index of Scale_M is applied to the image after M rounds of down sampling of the image by a factor of two.

Expression 1 is an example of computing a MS-SSIM index for an image from a first scale for the original resolution of the image (Scale 1) to a last scale (Scale_M) which is a result of M rounds of down sampling the original image by a factor of two. Expression 1 uses a scale for each round of down sampling from the original image (e.g., i=1) to the Mth scale (the last scale) (e.g., i=M) to calculate the MS_SIMM index for the image.

MS⁢-⁢SSIM=∏i=1M⁢⁢(SSIMiγi)(1)
where M=number of rounds, and γ=weighting factor. For example, for five rounds of down sampling by a factor of two (five subsamples), a SSIM is determined (calculated) for each round and then weighted based on a weighting factor, γ1, associated with the respective scale (i).

Expression 2 is an example of computing a weighting factor at each round of the down sampling (at each scale) (e.g., a weighting factor at each of i=1 to i=M.) The weighting factor may place an importance for each SIMM value for each round of the down sampling. Expression 2 is an example showing a normalization of the settings. Each setting is based on a common fixed viewing distance of a user to a display device and a fixed pixel density for each delivered image resolution (e.g., a fixed PPx per viewing distance).

Expression 3 is an example of computing a weighted MS-SSIM index for an image from a first scale for the original resolution of the image (Scale_1) to a last scale (Scale_M) which is a result of M rounds of down sampling the original image by a factor of two. A MS_SSIM module706included in an encoding mean opinion score module704may determine (compute) the weighted MS_SSIM index. The weighted MS_SSIM index may take into account a playback resolution, a display device pixel-width resolution (PPx), a pixel density, and an estimated viewing distance of the user to the display device when determining a weighting factor for each scale of the weighted MS-SIMM index.

For example, a weighting module708included in the encoding mean opinion score module704may determine (calculate) a weighting factor at each scale based on a contrast sensitivity at a corresponding spatial frequency (e.g., seeFIGS. 4 and 5, for example) for a display device of a playback device. The spatial frequency may be a function of the display device pixel-width resolution (PPx) and an estimated viewing distance of the user to the display device (e.g., seeFIGS. 1-3, for example). As described, a sensitivity of a visual system (e.g., a display device) may depend on a spatial frequency of a visual angle of a user to the display device. As shown inFIGS. 4 and 5, for example, human visual sensitivity (e.g., the contrast sensitivity404, the contrast sensitivity504) may peak at middle frequencies (e.g., at approximately four cycles per degree of visual angle) and may decrease in both a higher frequency direction and lower frequency direction. A spatial frequency may depend on a number of pixels per inch (PPi) (e.g., a display device pixel-width resolution (PPx)) and a viewing distance of a user to the display device.
Weighted MS-SSIM=SSIM(γi×δi)(3)
where M=number of rounds, γ=first weighting factor (as shown in Expression 2), and δ=second weighting factor (as shown in Expression 4). For example, for five rounds of down sampling (M=five) by a factor of two (five subsamples), a SSIM is determined (calculated) for each round and then weighted based on a first weighting factor and a second weighting factor associated with scale.
Fori=1 toi=M, δi=CSsf(4)
where CSsfis a contrast sensitivity value (CS) at a corresponding spatial frequency (sf). For example, seeFIGS. 4 and 5.

At the time of delivery of the encoded media content to a computing device (e.g., one or more of the computing devices608a-e) for display on a display device of the computing device (e.g., one or more of the display devices606a-e, respectively), the adaptive bitrate module634may determine a playback resolution, a display device pixel-width resolution (PPx), a pixel density, and established (e.g., estimated) viewing distance for the display device of the computing device. Because spatial frequency may be a function of a display device pixel density and a viewing distance of a user to the display device, a weighting (e.g., the second weighting factor δ) may be determined for each encoding resolution and for each subsample of the encoding resolution based on a contrast sensitivity at a corresponding spatial frequency. For example, referring to the graphs ofFIGS. 4 and 5, a second weighting factor may be determined at each subsample

For example, a low weighting may be applied to a 1080p encoding provided to a high pixel density display device of a first size because the high pixel density display device may be considered to have a high spatial frequency for relatively low contrast sensitivity. In another example, a weighting higher than the weighting applied to the 1080p encoding provided to the high pixel density display device may be applied to the 1080p encoding when provided to an HDTV. As such, a mean opinion score for a 1080p encoded image may differ dependent on the display device the image is viewed on. The use of a weighted MS-SIMM index as described herein may account for the display device differences (e.g., pixel density, display device pixel resolution) while also taking into account a typical viewing distance of the user to the display device.

Referring toFIGS. 1, 2, 3, and 6, in some implementations, an encoding delivery module714included in the adaptive bitrate module634may provide the same encoding to multiple different computing devices. Based on the characteristics of the computing device and the viewing distance of the user to the screen of the computing device, the same encoding may have a different perceived image quality. For example, the transmission speed module710may determine a transmission rate between the computing system612and the playback computing device722by way of the network720. The adaptive bitrate module634may use the determined transmission rate along with one or more characteristics of the playback computing device (e.g., a processor speed and/or computing capability) and one or more characteristics of the display device (e.g., pixel resolution, pixel density) to determine an encoding to send to the playback computing device722. In some implementations, the adaptive bitrate module634may provide the same encoding (e.g., the 1080p encoding604eto each of the computing devices608a-e. As described herein, the one or more characteristics of the playback computing device, the one or more characteristics of the display device of the computing device, and/or the viewing distance of a user to the display device of the computing device may result in different perceived image quality for the same encoding.

A frame rate module712included in the adaptive bitrate module634may determine a preferred (optimum) frame rate for delivery of streaming media to a playback computing device. In some implementations, a frame rate for an encoding may influence a perceived image quality for the encoding. For example, media content displayed at a high frame rate may be desired for interactive streaming media content (e.g., video games). In some cases, however, if one or more characteristics of a computing device and/or one or more characteristics of a display device included in the computing device are not capable of providing and displaying, respectively, the streaming media content at the desired frame rate, the perceived image quality of the streaming media content may be reduced as compared to a computing device that is capable of providing the streaming media content at the desired frame rate. In some implementations, however, in cases where one or more characteristics of a computing device and/or one or more characteristics of the display device included in the computing device may not be capable of providing and displaying, respectively, streaming media content at a first frame rate, dependent on the content of the streaming media content, a computing system may provide the streaming media content to the playback computing device at a second frame rate, less than the first frame rate, but sufficient to properly display the streaming media content resulting in favorable perceived image quality for the streaming media.

FIG. 8is a diagram showing an example graph800of predicted mean opinion scores based on an encoding mean opinion score (encoding MOS802) verses a mean opinion score based on information provided by a user (viewer) (Human MOS804). The example graph800shows that a predicted encoding mean opinion score as compared to a mean opinion score based on information provided by a user is reasonable and effective. For example, a Spearman ranking may be equal to 0.873 and a Pearson linearity may be equal to 0.872.

FIG. 9is a block diagram of an example system900that includes modules for use in measuring image quality based on an image quality metric. Referring toFIGS. 6 and 7, modules920may include the encoding module630, the adaptive bitrate module634, and the encoding mean opinion score module704. The modules920may include a content receiving module922, a communication module924, a data gathering module936, and a content delivery module938.

Although illustrated as separate elements, one or more of modules920inFIG. 9may represent portions of a single module or application.

Referring toFIG. 6andFIG. 7, the content receiving module922may receive media content632(which may be compressed or transcoded) from the content provider computing device626. The content receiving module922may store the received media content632in the received content storage repository616. The communication module924may facilitate communications between the system900and a content provider computing device (e.g., the content provider computing device626). The communication module924may determine and/or provide a bandwidth of a communication connection between the system900and a computing device to the adaptive bitrate module634for use in determining an encoding and transmission bitrate for delivery of the encoded media content to the computing device. The communication module924may determine and/or provide a bandwidth of a communication connection between the system900and a computing device to the content delivery module938. The content delivery module938may provide (transmit) the selected encoding of the media content to the computing device. The encoding mean opinion score module704may determine (calculate) an encoding mean opinion score that is based on a MS_SIMM index and a weighting as determined (calculated) by the MS-SSIM module706and the weighting module708, respectively.

The data gathering module936may gather information and data for use in determining (calculating) weightings, reference metrics, MS-SIMM indexes, and encoding mean opinion scores. For example, information and data associated with a perceived quality of viewed media content associated with delivered resolutions (encodings), delivered transmission speeds, and delivered frame rates may be used to determine (calculate) an encoding mean opinion score. For example, a computing system may deliver a video to a computing device of a user at a particular resolution (encoding), transmission speed, and frame rate. The user may provide a score for a perceived quality of the delivered video which may later be used by the encoding mean opinion score module704to determine (generate) an encoding mean opinion score for the video delivered to the computing device at the particular video resolution, transmission speed, and frame rate. The generated encoded mean opinion score may be stored in a table for use in predicting encoding mean opinion scores for subsequent media content for delivery to the computing device as disclosed herein.

In certain embodiments, one or more of modules920inFIG. 9may represent one or more software applications or programs that, when executed by a computing system, may cause the computing system to perform one or more tasks. As illustrated inFIG. 9, example system900may also include one or more memory devices, such as memory910. Memory910generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and/or computer-readable instructions. In one example, memory910may store, load, and/or maintain one or more of modules920. Examples of memory910include, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives, caches, variations or combinations of one or more of the same, and/or any other suitable storage memory.

As illustrated inFIG. 9, example system900may also include one or more physical processors, such as physical processor930. Physical processor930generally represents any type or form of hardware-implemented processing unit capable of interpreting and/or executing computer-readable instructions. In one example, physical processor930may access and/or modify one or more of modules920stored in memory910. Additionally, or alternatively, physical processor930may execute one or more of modules920. Examples of physical processor DD30 include, without limitation, microprocessors, microcontrollers, Central Processing Units (CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcore processors, Application-Specific Integrated Circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, and/or any other suitable physical processor.

As illustrated inFIG. 9, example system900may also include one or more additional elements940. The additional elements940generally represent any type or form of hardware and/or software. In one example, physical processor930may access and/or modify one or more of the additional elements940.

The additional elements940may be included in one or more repositories. The one or more repositories may be memory (e.g., the memory910). The one or more repositories may be databases. In some implementations, the additional elements940may be included (part of) the system900. In some implementations, the additional elements940may be external to the system900and accessible by the system900. Referring toFIG. 6andFIG. 7, the additional elements940may include the received content storage repository616and the encoded content storage638. The encoded content storage638may store encodings604a-f.

FIG. 10illustrates an exemplary network environment1000in which aspects of the present disclosure may be implemented. The network environment1000may include one or more computing devices (e.g., the content provider computing device636, the playback computing device722), the network720, and a server1006. The playback computing device722may represent any one of the playback computing devices608a-e. The playback computing device722may represent any one of the mobile computing device106, the desktop computing device206, and the HDTV306.

In one example, the server1006may host a system for receiving media content, determining (calculating) weighted metrics and encoding mean opinion scores for the media content, and delivering an encoding of the media content to a playback computing device. For example, the server1006may host all or part of the system900as shown inFIG. 9. In this example, the server1006may include a physical processor1060that may be one or more general-purpose processors that execute software instructions. The server1006may include a data storage subsystem that includes a memory1010which may store software instructions, along with data (e.g., input and/or output data) processed by execution of those instructions. Referring toFIG. 9, the memory1010may include the modules920.

The server1006may include additional elements1040. ReferringFIG. 9, the additional elements1040may include all or part of the additional elements940. In some implementations, all or part of the additional elements1040may be external to the server1006and the playback computing device722and may be accessible by the server1006either directly (a direct connection) or by way of the network720.

The content provider computing device636may represent a client device or a user device, such a desktop computer, laptop computer, tablet device, smartphone, or other computing device, examples of which are included herein. The content provider computing device636may include a physical processor (e.g., physical processor1020), which may represent a single processor or multiple processors, and one or more memory devices (e.g., memory1024), which may store instructions (e.g., software applications) and/or data in one or more modules1026. The modules1026may store software instructions, along with data (e.g., input and/or output data) processed by execution of those instructions.

The content provider computing device636may be (represent) a computing device of a user. The content provider computing device636may include storage for the media content632(e.g., content storage1034) obtained, created, and/or generated by the user. In some implementations, media content stored in the media content632may be accessed by a content application1030. The content application1030may include hardware and/or software for displaying the media content on a display device1022included in the content provider computing device636. In addition, or in the alternative, the content application1030may include hardware and/or software for providing media content to the server1006by way of the network720. A communication module1028may include hardware and/or software for establishing a connection to the server1006by way of the network720, for example, by interfacing with the communication module924included in the modules920. In some implementations, the content application1030may include hardware and/or software for providing media content to a transcoder602. The transcoder602may include hardware and/or software for transcoding and/or compressing the media content for subsequent delivery to the server1006. In some implementations, the transcoded and/or compressed media content may be stored in the media content632for later delivery to the server1006. One or more audio device(s)1036may include hardware and/or software for playing audio media content (e.g., one or more speakers) and/or for recording audio media content (e.g., one or more microphones).

The content provider computing device636may be communicatively coupled to the server1006through the network720. The network720may be any communication network, such as the Internet, a Wide Area Network (WAN), or a Local Area Network (LAN), and may include various types of communication protocols and physical connections.

The playback computing device722may represent a client device or a user device, such a desktop computer, laptop computer, tablet device, smartphone, or other computing device as disclosed herein. In addition, or in the alternative, the playback computing device722may represent a smart TV, an HDTV, a digital display device, an electronic visual display, or any type of computing device or display device that may communicate with the server1006by way of the network720. The playback computing device722may include a physical processor (e.g., physical processor1070), which may represent a single processor or multiple processors, and one or more memory devices (e.g., memory1044), which may store instructions (e.g., software applications) and/or data in one or more modules1046. The modules1046may store software instructions, along with data (e.g., input and/or output data) processed by execution of those instructions.

The playback computing device722may be (represent) a computing device of a user. The playback computing device722may receive encoded media content from the server1006by way of the network720for display on the display device724. In some implementations, the playback computing device722may store the received encoded media content in a content storage repository1056for later playing (displaying) on the display device724. In some implementations, the received media content may be played (displayed) on the display device724as it is received from the server1006(e.g., the received media content is streamed to the display device724). A playback application1050may include hardware and/or software for displaying (playing) the received media content on the display device724for viewing by the user.

In some implementations, the playback application1050may include hardware and/or software for interpreting transcoded and/or compress media content when providing the media content for displaying (playing) on the display device724. One or more audio device(s)1054may include hardware and/or software for playing audio media content (e.g., one or more speakers) and/or for recording audio media content (e.g., one or more microphones). An orientation module1052may include hardware and/or software for determining an orientation (e.g., vertical, horizontal, portrait, landscape) of the playback computing device722. For example, the orientation module1052may include one or more sensors that may include, but are not limited to, accelerometers, gyroscopes, magnetometers, and other suitable types of sensors that may be used to detect and/or determine an orientation of the playback computing device722.

A communication module1048may include hardware and/or software for establishing a connection to the server1006by way of the network720, for example, by interfacing with the communication module924included in the modules920. The playback computing device722may be communicatively coupled to server1006through the network720. The network720may be any communication network, such as the Internet, a Wide Area Network (WAN), or a Local Area Network (LAN), and may include various types of communication protocols and physical connections.

FIG. 11is a flow diagram of an exemplary computer-implemented method1100for measuring image quality based on an image quality metric. The steps shown inFIG. 11may be performed by any suitable computer-executable code and/or computing system, including the system(s) illustrated inFIGS. 6, 7, 9, and 10. In one example, each of the steps shown inFIG. 11may represent an algorithm whose structure includes and/or is represented by multiple sub-steps, examples of which will be provided in greater detail below.

As illustrated inFIG. 11, at step1102one or more of the systems described herein may encode media content into a plurality of encoded media files, each encoded media file having an encoded resolution and at least one associated full reference metric. For example, the encoding module630may encode media content into one or more progressive scanned encodings (e.g., encodings604a-e) and into a 4k resolution encoding604f.

The systems described herein may perform step1102in a variety of ways. In one example, the encoding module630may encode media content into one or more progressive scanned encodings (e.g., encodings604a-e) and into a 4k resolution encoding604ffor storage in the encoded content storage638.

As illustrated inFIG. 11, at step1104one or more of the systems described herein may identify one of the plurality of encoded media files to provide to a computing device based on at least a playback resolution of a display device included in the computing device. For example, an adaptive bitrate module634may determine a playback resolution of a display device (e.g., the display device724) included in the playback computing device722.

The systems described herein may perform step1104in a variety of ways. In one example, the adaptive bitrate module634, based on determining a playback resolution of a display device (e.g., the display device724) included in the playback computing device722, may identify one of the encodings604a-fto provide to the playback computing device722as described herein.

As illustrated inFIG. 11, at step1106one or more of the systems described herein may weight a measurement of a quality of the encoded media file based on the at least one associated full reference metric and on at least one characteristic associated with the display device. For example, the weighting module708included in the encoding mean opinion score module704may determine (calculate) a weight for use in weighting the measurement of the quality of the encoded media file delivered to the playback computing device722.

The systems described herein may perform step1106in a variety of ways. In one example, the MS-SSIM module706may determine (calculate) a weighted MS-SSIM index for an encoding based on a full reference metric associated with the encoding and weightings determined (calculated) by the weighting module708. The encoding mean opinion score module704may apply the weighted MS-SSIM index determined (calculated) by the MS-SSIM module706to the encoding as a score of a perceived image quality for the encoding that is weighted based on one or more characteristics of the display device724, one or more characteristics of the playback computing device722, and a viewing distance of a user to the display device724, as disclosed herein.

EXAMPLE EMBODIMENTS

Example 1: A computer-implemented method may include encoding media content into a plurality of encoded media files, each encoded media file having an encoded resolution and at least one associated full reference metric, identifying one of the plurality of encoded media files to provide to a computing device based on at least a playback resolution of a display device included in the computing device, and weighting a measurement of a quality of the encoded media file based on the at least one associated full reference metric and on at least one characteristic associated with the display device.

Example 2: The computer-implemented method of Example 1, where a characteristic associated with the display device may be a pixel density of the display device.

Example 3: The computer-implemented method of any of Examples 1 and 2, where a characteristic associated with the display device may be a viewing distance of a user to the display device.

Example 4: The computer-implemented method of any of Examples 1-3, further including performing the measurement of the quality of the encoded media file at multiple scales, and where, at each scale, weighting the measurement of the quality of the encoded media file may further include calculating a spatial frequency associated with the computing device, determining a contrast sensitivity associated with the spatial frequency, and weighting the measurement at the scale based on the contrast sensitivity.

Example 5: The computer-implemented method of any of Examples 1-4, where the at least one associated full reference metric may be a structural similarity index.

Example 6: The computer-implemented method of Example 5, further including performing the measurement of the quality of the encoded media file at multiple scales, each scale being a down sample of the encoded resolution, and generating a multi-scale structural similarity index.

Example 7: The computer-implemented method of any of Examples 5 and 6, where a spatial frequency may be associated with each scale of the multiple scales, and where weighting the measurement of the quality of the encoded media file may include weighting each scale of the multi-scale structural similarity index based on a contrast sensitivity for the spatial frequency associated with the scale.

Example 8: The computer-implemented method of any of Examples 1-7, where identifying the encoded media file to provide to the computing device may be further based on a frame rate for playback of the encoded media file on the display device.

Example 9: A system may include at least one physical processor, and physical memory including computer-executable instructions that, when executed by the physical processor, cause the physical processor to encode media content into a plurality of encoded media files, each encoded media file having an encoded resolution and at least one associated full reference metric, identify one of the plurality of encoded media files to provide to a computing device based on at least a playback resolution of a display device included in the computing device, and weight a measurement of a quality of the encoded media file based on the at least one associated full reference metric and on at least one characteristic associated with the display device.

Example 10: The system of Example 9, where a characteristic associated with the display device may be a pixel density of the display device.

Example 11: The system of any of Examples 9 and 10, where a characteristic associated with the display device may be a viewing distance of a user to the display device.

Example 12: The system of any of Examples 9-11, where the computer-executable instructions further cause the physical processor to perform the measurement of the quality of the encoded media file at multiple scales, and where, at each scale, the computer-executable instructions that cause the physical processor to weight the measurement of the quality of the encoded media file may further include instructions that cause the physical processor to calculate a spatial frequency associated with the computing device, determine a contrast sensitivity associated with the spatial frequency, and weight the measurement at the scale based on the contrast sensitivity.

Example 13: The system of any of Examples 9-12, where the at least one associated full reference metric may be a structural similarity index.

Example 14: The system of Example 13, where the computer-executable instructions further cause the physical processor to perform the measurement of the quality of the encoded media file at multiple scales, each scale being a down sample of the encoded resolution, and generate a multi-scale structural similarity index.

Example 15: The system of any of Examples 13 and 14, where a spatial frequency may be associated with each scale of the multiple scales, and where the computer-executable instructions that cause the physical processor to weight the measurement of the quality of the encoded media file may further include instructions that cause the physical processor to weight each scale of the multi-scale structural similarity index based on a contrast sensitivity for the spatial frequency associated with the scale.

Example 16: The system of any of Examples 9-15, where identifying the encoded media file to provide to the computing device may be further based on a frame rate for playback of the encoded media file on the display device.

Example 17: A non-transitory computer-readable medium may include one or more computer-executable instructions that, when executed by at least one processor of a computing system, may cause the computing system to encode media content into a plurality of encoded media files, each encoded media file having an encoded resolution and at least one associated full reference metric, identify one of the plurality of encoded media files to provide to a computing device based on at least a playback resolution of a display device included in the computing device, and weight a measurement of a quality of the encoded media file based on the at least one associated full reference metric and on at least one characteristic associated with the display device.

Example 18: The non-transitory computer-readable medium of Example 17, where a characteristic associated with the display device may be a pixel density of the display device.

Example 19: The non-transitory computer-readable medium of any of Examples 17 and 18, where a characteristic associated with the display device may be a viewing distance of a user to the display device.

Example 20: The cyst non-transitory computer-readable medium of any of Examples 17-19, where one or more of the computer-executable instructions further cause the computing system to perform the measurement of the quality of the encoded media file at multiple scales, and where, at each scale, the one or more computer-executable instructions that cause the computing system to weight the measurement of the quality of the encoded media file may further cause the computing system to calculate a spatial frequency associated with the computing device, determine a contrast sensitivity associated with the spatial frequency, and weight the measurement at the scale based on the contrast sensitivity.

Turning toFIG. 12, augmented-reality system1200generally represents a wearable device dimensioned to fit about a body part (e.g., a head) of a user. As shown inFIG. 12, system1200may include a frame1202and a camera assembly1204that is coupled to frame1202and configured to gather information about a local environment by observing the local environment. Augmented-reality system1200may also include one or more audio devices, such as output audio transducers1208(A) and1208(B) and input audio transducers1210. Output audio transducers1208(A) and1208(B) may provide audio feedback and/or content to a user, and input audio transducers1210may capture audio in a user's environment.

As shown, augmented-reality system1200may not necessarily include an NED positioned in front of a user's eyes. Augmented-reality systems without NEDs may take a variety of forms, such as head bands, hats, hair bands, belts, watches, wrist bands, ankle bands, rings, neckbands, necklaces, chest bands, eyewear frames, and/or any other suitable type or form of apparatus. While augmented-reality system1200may not include an NED, augmented-reality system1200may include other types of screens or visual feedback devices (e.g., a display screen integrated into a side of frame1202).

The embodiments discussed in this disclosure may also be implemented in augmented-reality systems that include one or more NEDs. For example, as shown inFIG. 13, augmented-reality system1300may include an eyewear device1302with a frame1310configured to hold a left display device1315(A) and a right display device1315(B) in front of a user's eyes. Display devices1315(A) and1315(B) may act together or independently to present an image or series of images to a user. While augmented-reality system1300includes two displays, embodiments of this disclosure may be implemented in augmented-reality systems with a single NED or more than two NEDs.

In some embodiments, augmented-reality system1300may include one or more sensors, such as sensor1340. Sensor1340may generate measurement signals in response to motion of augmented-reality system1300and may be located on substantially any portion of frame1310. Sensor1340may represent a position sensor, an inertial measurement unit (IMU), a depth camera assembly, or any combination thereof. In some embodiments, augmented-reality system1300may or may not include sensor1340or may include more than one sensor. In embodiments in which sensor1340includes an IMU, the IMU may generate calibration data based on measurement signals from sensor1340. Examples of sensor1340may include, without limitation, accelerometers, gyroscopes, magnetometers, other suitable types of sensors that detect motion, sensors used for error correction of the IMU, or some combination thereof. Augmented-reality system1300may also include a microphone array with a plurality of acoustic transducers1320(A)-1320(J), referred to collectively as acoustic transducers1320. Acoustic transducers1320may be transducers that detect air pressure variations induced by sound waves. Each acoustic transducer1320may be configured to detect sound and convert the detected sound into an electronic format (e.g., an analog or digital format). The microphone array inFIG. 2may include, for example, ten acoustic transducers:1320(A) and1320(B), which may be designed to be placed inside a corresponding ear of the user, acoustic transducers1320(C),1320(D),1320(E),1320(F),1320(G), and1320(H), which may be positioned at various locations on frame1310, and/or acoustic transducers1320(1) and1320(J), which may be positioned on a corresponding neckband1305.

In some embodiments, one or more of acoustic transducers1320(A)-(F) may be used as output transducers (e.g., speakers). For example, acoustic transducers1320(A) and/or1320(B) may be earbuds or any other suitable type of headphone or speaker.

The configuration of acoustic transducers1320of the microphone array may vary. While augmented-reality system1300is shown inFIG. 13as having ten acoustic transducers1320, the number of acoustic transducers1320may be greater or less than ten. In some embodiments, using higher numbers of acoustic transducers1320may increase the amount of audio information collected and/or the sensitivity and accuracy of the audio information. In contrast, using a lower number of acoustic transducers1320may decrease the computing power required by an associated controller1350to process the collected audio information. In addition, the position of each acoustic transducer1320of the microphone array may vary. For example, the position of an acoustic transducer1320may include a defined position on the user, a defined coordinate on frame1310, an orientation associated with each acoustic transducer1320, or some combination thereof.

Acoustic transducers1320(A) and1320(B) may be positioned on different parts of the user's ear, such as behind the pinna or within the auricle or fossa. Or, there may be additional acoustic transducers1320on or surrounding the ear in addition to acoustic transducers1320inside the ear canal. Having an acoustic transducer1320positioned next to an ear canal of a user may enable the microphone array to collect information on how sounds arrive at the ear canal. By positioning at least two of acoustic transducers1320on either side of a user's head (e.g., as binaural microphones), augmented-reality device1300may simulate binaural hearing and capture a 3D stereo sound field around about a user's head. In some embodiments, acoustic transducers1320(A) and1320(B) may be connected to augmented-reality system1300via a wired connection1330, and in other embodiments, acoustic transducers1320(A) and1320(B) may be connected to augmented-reality system1300via a wireless connection (e.g., a Bluetooth connection). In still other embodiments, acoustic transducers1320(A) and1320(B) may not be used at all in conjunction with augmented-reality system1300.

Acoustic transducers1320on frame1310may be positioned along the length of the temples, across the bridge, above or below display devices1315(A) and1315(B), or some combination thereof. Acoustic transducers1320may be oriented such that the microphone array is able to detect sounds in a wide range of directions surrounding the user wearing the augmented-reality system1300. In some embodiments, an optimization process may be performed during manufacturing of augmented-reality system1300to determine relative positioning of each acoustic transducer1320in the microphone array.

In some examples, augmented-reality system1300may include or be connected to an external device (e.g., a paired device), such as neckband1305. Neckband1305generally represents any type or form of paired device. Thus, the following discussion of neckband1305may also apply to various other paired devices, such as charging cases, smart watches, smart phones, wrist bands, other wearable devices, hand-held controllers, tablet computers, laptop computers and other external compute devices, etc.

As shown, neckband1305may be coupled to eyewear device1302via one or more connectors. The connectors may be wired or wireless and may include electrical and/or non-electrical (e.g., structural) components. In some cases, eyewear device1302and neckband1305may operate independently without any wired or wireless connection between them. WhileFIG. 13illustrates the components of eyewear device1302and neckband1305in example locations on eyewear device1302and neckband1305, the components may be located elsewhere and/or distributed differently on eyewear device1302and/or neckband1305. In some embodiments, the components of eyewear device1302and neckband1305may be located on one or more additional peripheral devices paired with eyewear device1302, neckband1305, or some combination thereof.

Neckband1305may be communicatively coupled with eyewear device1302and/or to other devices. These other devices may provide certain functions (e.g., tracking, localizing, depth mapping, processing, storage, etc.) to augmented-reality system1300. In the embodiment ofFIG. 13, neckband1305may include two acoustic transducers (e.g.,1320(1) and1320(J)) that are part of the microphone array (or potentially form their own microphone subarray). Neckband1305may also include a controller1325and a power source1335.

Acoustic transducers1320(1) and1320(J) of neckband1305may be configured to detect sound and convert the detected sound into an electronic format (analog or digital). In the embodiment ofFIG. 13, acoustic transducers1320(1) and1320(J) may be positioned on neckband1305, thereby increasing the distance between the neckband acoustic transducers1320(1) and1320(J) and other acoustic transducers1320positioned on eyewear device1302. In some cases, increasing the distance between acoustic transducers1320of the microphone array may improve the accuracy of beamforming performed via the microphone array. For example, if a sound is detected by acoustic transducers1320(C) and1320(D) and the distance between acoustic transducers1320(C) and1320(D) is greater than, e.g., the distance between acoustic transducers1320(D) and 1320(E), the determined source location of the detected sound may be more accurate than if the sound had been detected by acoustic transducers1320(D) and1320(E).

Controller1325of neckband1305may process information generated by the sensors on neckband1305and/or augmented-reality system1300. For example, controller1325may process information from the microphone array that describes sounds detected by the microphone array. For each detected sound, controller1325may perform a direction-of-arrival (DOA) estimation to estimate a direction from which the detected sound arrived at the microphone array. As the microphone array detects sounds, controller1325may populate an audio data set with the information. In embodiments in which augmented-reality system1300includes an inertial measurement unit, controller1325may compute all inertial and spatial calculations from the IMU located on eyewear device1302. A connector may convey information between augmented-reality system1300and neckband1305and between augmented-reality system1300and controller1325. The information may be in the form of optical data, electrical data, wireless data, or any other transmittable data form. Moving the processing of information generated by augmented-reality system1300to neckband1305may reduce weight and heat in eyewear device1302, making it more comfortable to the user.

Power source1335in neckband1305may provide power to eyewear device1302and/or to neckband1305. Power source1335may include, without limitation, lithium ion batteries, lithium-polymer batteries, primary lithium batteries, alkaline batteries, or any other form of power storage. In some cases, power source1335may be a wired power source. Including power source1335on neckband1305instead of on eyewear device1302may help better distribute the weight and heat generated by power source1335.

As noted, some artificial-reality systems may, instead of blending an artificial reality with actual reality, substantially replace one or more of a user's sensory perceptions of the real world with a virtual experience. One example of this type of system is a head-worn display system, such as virtual-reality system1400inFIG. 14, that mostly or completely covers a user's field of view. Virtual-reality system1400may include a front rigid body1402and a band1404shaped to fit around a user's head. Virtual-reality system1400may also include output audio transducers1406(A) and1406(B). Furthermore, while not shown inFIG. 14, front rigid body1402may include one or more electronic elements, including one or more electronic displays, one or more inertial measurement units (IMUS), one or more tracking emitters or detectors, and/or any other suitable device or system for creating an artificial reality experience.

Artificial-reality systems may also include one or more input and/or output audio transducers. In the examples shown inFIGS. 12 and 14, output audio transducers1208(A),1208(B),1406(A), and1406(B) may include voice coil speakers, ribbon speakers, electrostatic speakers, piezoelectric speakers, bone conduction transducers, cartilage conduction transducers, and/or any other suitable type or form of audio transducer. Similarly, input audio transducers1210may include condenser microphones, dynamic microphones, ribbon microphones, and/or any other type or form of input transducer. In some embodiments, a single transducer may be used for both audio input and audio output.