Patent Description:
Many kinds of autonomous vehicles (e.g., cars, aircraft, watercraft) need to determine which direction is up and which is down. Some vehicles make this determination using global positioning system (GPS) data and/or inertial navigation system (INS) data to determine where the sky and/or the horizon relative to the vehicle. However, visual imagery can also be used to determine where the sky and/or the horizon is relative to the vehicle. For example, a sky segmentation process may be applied to an image or video frame to determine where the sky and/or the horizon is relative to the vehicle. Sky segmentation uses a machine-learning model to associate pixels in an image or video frame with the sky.

Sky segmentation methods fall generally into one of three categories. One category uses color or intensity information to divide the image into two regions (e.g., ground and sky). Another category uses gradient-based methods to detect the horizon as the longest horizontal edge in the image. Another category uses deep learning semantic segmentation to segment sky regions.

A method for segmenting an image into a sky region and a ground region is disclosed. The method includes receiving or identifying a first image. The method also includes determining a distribution based at least partially upon an intensity of each pixel in the first image. The method also includes determining that the distribution is bimodal. The method also includes dividing the first image to produce a second image in response to determining that the distribution is bimodal. The second image includes a plurality of first pixels and a plurality of second pixels. The method also includes determining that a horizon is defined between the plurality of first pixels and the plurality of second pixels.

A computing system is also disclosed. The computing system includes one or more processors and a memory system. The memory system includes one or more non-transitory computer-readable media storing instructions that, when executed by at least one of the one or more processors, cause the computing system to perform operations. The operations include identifying a first image. The operations also include identifying a number of edges in the first image. The operations also include determining a distribution based at least partially upon an intensity of each pixel in the first image when the number of edges is greater than a predetermined threshold. The operations also include determining that the distribution approximates a bimodal mixture of Gaussian distributions. The operations also include in response to determining that the distribution approximates the bimodal mixture of Gaussian distributions, dividing the first image to produce a second image. The second image includes a plurality of first pixels corresponding to a ground region and a plurality of second pixels corresponding to a sky region. The operations also include determining that a horizon is defined between the first pixels and the second pixels.

In another example, the operations include identifying a first image. The image is captured by a camera on an aircraft in flight. The operations also include identifying a number of edges in the first image. The operations also include determining a histogram based at least partially upon an intensity of each pixel in the first image when the number of edges is greater than a predetermined threshold. The operations also include determining that the histogram more closely approximates a bimodal mixture of Gaussian distributions than a unimodal Gaussian distribution. Determining that the histogram more closely approximates the bimodal mixture of Gaussian distributions includes determining a first error between the histogram and the bimodal mixture of Gaussian distributions that corresponds to the histogram. The determination that the histogram more closely approximates the bimodal mixture of Gaussian distributions is based at least partially upon the first error. In response to determining that the histogram more closely approximates the bimodal mixture of Gaussian distributions, the operations also include dividing the first image to produce a second image. The second image includes a plurality of first pixels corresponding to a ground region and a plurality of second pixels corresponding to a sky region. The first pixels have a first color, and the second pixels have a second, different color. The operations also include filtering the second image to produce a third image. The operations also include determining that a horizon is defined between the first pixels and the second pixels in the third image. The operations also include combining the first image and at least a portion of the third image to produce a fourth image. The fourth image also includes the horizon.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate aspects of the present teachings and together with the description, serve to explain the principles of the present teachings.

It should be noted that some details of the figures have been simplified and are drawn to facilitate understanding rather than to maintain strict structural accuracy, detail, and scale.

Reference will now be made in detail to the present teachings, examples of which are illustrated in the accompanying drawings. In the drawings, like reference numerals have been used throughout to designate identical elements. In the following description, reference is made to the accompanying drawings that form a part thereof, and in which is shown by way of illustration specific examples of practicing the present teachings. The following description is, therefore, merely exemplary.

The system and method disclosed herein may be used to visually detect and localize sky regions within an image. This solution may be employed in an air-to-air visual detection system for detect-and-avoid efforts. If both sky and ground are present in an image, the most different regions may be the sky region and the ground region. Such image may be taken from aircraft in air. This has proven to hold in most situations because the aircraft is high above the ground, and the ground generally loses most of the fine detail which would contribute to higher intraclass variance. If only sky is present in the image, few strong edges will be present. Sometimes certain types of clouds can contribute to strong edges, but in general, the sky is free of strong edges. If only ground is present in the image, an intensity representation (e.g., a histogram) may be generally unimodal. Thus, the problem may include determining the image representation, determining if it is unimodal or multimodal, and then if it is multimodal, determining an optimal intensity threshold for dividing the image into a first (e.g., dark) region and a second (e.g., light) region.

The system and method disclosed herein provide increased speed and accuracy over conventional systems and methods. For example, one method, deep semantic segmentation, does not run in real-time. Thus, there is a large time investment required to create the datasets needed for ground truth, as well as actually training the algorithm. In contrast, the system and method disclosed herein do not require these time investments. In addition, the system and method disclosed herein result in an intuitive definition of multimodality of a discrete function. The method may include using a Canny edge detector to determine how many strong edges are present in the image. If few edges are found, the image is determined to be all sky and the function returns. A histogram of the grayscale image is then determined. The method may then include determining if the histogram is unimodal or multimodal (e.g., bimodal). Because the histogram may represent a discrete, discontinuous function, the usual definition of a mode as a local maximum may not apply. The method may then include using the Levenberg-Marquadt algorithm to approximate the histogram with both unimodal and bimodal Gaussian distributions. The method may then use the error between the actual histogram values and the distributions to classify the histogram as either bimodal or unimodal. If the histogram is determined to be bimodal, then Otsu's thresholding method may be used to optimally divide the image into sky and ground regions, and this binary output is returned. Otherwise, the image is determined to be all ground and the function returns. If the function was determined to be bimodal, then holes in the sky region and ground region are removed through morphology.

Using Otsu thresholding to segment images containing both sky and ground has not been previously used in situations where the image contains only sky or only ground. In these situations, Otsu's method will still split the image into dark and light regions, but these regions will not correspond to sky and ground. In case, the image histograms are discrete and not smooth, the traditional definition of a mode as a local maximum may not capture the overall trend of the data. The method of approximating the distribution with both a unimodal and bimodal Gaussian function is fast and still captures the trend of the data. The method also sets bounds on the feasible magnitudes of the Gaussian functions. Otherwise, the bimodal function may just set one of the modal regions to have a magnitude of <NUM>, and effectively become a unimodal function. The feasible region has been optimized to best capture the trends of the data.

<FIG> illustrates a schematic view of an aircraft <NUM> in flight, according to an example. The aircraft <NUM> may be or include an airplane, a helicopter, an unmanned aerial vehicle (e.g., a drone), a spacecraft, or the like. The aircraft <NUM> may include a camera <NUM>. The camera <NUM> may be coupled the aircraft <NUM> and/or positioned within the aircraft <NUM>. The camera <NUM> may be configured to capture one or more images. The camera <NUM> may also or instead be configured to capture a video. In one example, the camera <NUM> may be configured to capture continual stream of images over time (i.e. a video), and the images may be still frames taken from the video.

The images and/or video may be transmitted to a computing system <NUM> on the aircraft <NUM>. In another example, the computing system <NUM> may be located on the ground (e.g., in a control station) in communication with an on-board computing system. The computing system <NUM> may include one or more processors and a memory system. The memory system may include one or more non-transitory computer-readable media storing instructions that, when executed by at least one of the one or more processors, cause the computing system to perform operations. An example of the operations is provided in <FIG>.

<FIG> illustrates a flowchart of a method <NUM> for segmenting an image into a sky region and a ground region, according to an example. An illustrative order of the method <NUM> is described below. One or more steps of the method <NUM> may be performed in a different order, repeated, or omitted. As indicated below, one or more steps of the method <NUM> may be performed by the computing system <NUM>.

The method <NUM> may include receiving or identifying a first image, as at <NUM>. An example of the first image <NUM> is shown in <FIG>. The first image <NUM> may be captured by the camera <NUM> and transmitted to and received/identified by the computing system <NUM>. The camera <NUM> may be in an upright position and pointed forward such that the first image <NUM> is in front of the aircraft <NUM>. Thus, the first image <NUM> may include at least a portion of the flight path of the aircraft <NUM>. The first image <NUM> may include a ground region <NUM>, a sky region <NUM>, or both. In this particular example, the first image <NUM> may also include an occlusion (e.g., from nearby objects on the ground) <NUM>. The first image <NUM> may be in color, and it may be converted to grayscale and/or binary (e.g., black and white).

The method <NUM> may also include detecting a plurality of edges (three are identified: <NUM>, <NUM>, <NUM>) in the first image <NUM> to produce an edge-detected first image <NUM>, as at <NUM>. <FIG> shows the edge-detected first image <NUM> including edges <NUM>, <NUM>, <NUM>. As used herein, an "edge" refers to a point or line (e.g., one or more pixels) in an image where the brightness changes sharply (e.g., by more than a predetermined threshold). Although the edge-detected first image <NUM> includes more than three edges, for the sake of simplicity, only three are identified in <FIG>. This step may be performed by the computing system <NUM>. More particularly, the computing system <NUM> may use canny edge detection (e.g., with low hysteresis parameters) or Laplacian edge detection to detect the edges <NUM>, <NUM>, <NUM>. The detection may be based at least partially upon a sensitivity threshold of the intensity of the pixels in the image <NUM> and/or the image <NUM>. For example, the sensitivity threshold may be increased (e.g., by a user) to detect more edges in the image <NUM> and/or the image <NUM>, or the sensitivity threshold may be decreased to detect fewer edges in the image <NUM> and/or the image <NUM>. The image <NUM> may be binary, and pixels representing an edge <NUM>, <NUM>, <NUM> may have a value of zero (e.g., white), and pixels that do not represent an edge <NUM>, <NUM>, <NUM> may have a value of one (e.g., black), or vice versa.

The method <NUM> may also include determining whether a number of the edges <NUM>, <NUM>, <NUM> in the image <NUM> is greater than a predetermined amount/threshold, as at <NUM>. This step may be performed by the computing system <NUM>. When the number of edges <NUM>, <NUM>, <NUM> is less than the predetermined amount/threshold, it may be determined that the image <NUM> and/or the image <NUM> includes no ground region <NUM> and all sky region <NUM>. The method <NUM> may then loop back around to step <NUM>. When the number of edges <NUM>, <NUM>, <NUM> is greater than the predetermined amount/threshold, it may be determined that the image <NUM> and/or the image <NUM> includes the ground region <NUM>. For example, it may be determined that the image <NUM> and/or the image <NUM> includes a combination of the ground region <NUM> and the sky region <NUM>. The method <NUM> may then proceed as described below.

The method <NUM> may also include determining a distribution <NUM> based at least partially upon the image <NUM> and/or the image <NUM>, as at <NUM>. This step may be performed by the computing system <NUM>. More particularly, in response to the number of edges <NUM>, <NUM>, <NUM> being greater than the predetermined amount/threshold, the computing system <NUM> may determine the distribution <NUM> based at least partially upon an intensity of one or more (e.g., each) pixel in the image <NUM> and/or the image <NUM>. This is shown in <FIG>. The distribution <NUM> may be or include a histogram. The first image <NUM> may be stored in an <NUM>-bit format, meaning that there are <NUM> intensity possibilities for each pixel. Thus, the X-axis of the distribution <NUM> may go from <NUM>-<NUM>. The Y-axis may go from <NUM>-<NUM>.

The method <NUM> may also include determining whether the distribution <NUM> is unimodal or bimodal, as at <NUM>. This step may be performed by the computing system <NUM>. This step may include comparing the distribution <NUM> to a unimodal Gaussian distribution <NUM> that corresponds to the distribution <NUM>. This is also shown in <FIG>. Then, a first error may be determined between the distribution <NUM> and the unimodal Gaussian distribution <NUM>. In this particular example, the first error is <NUM>. This step may also include comparing the distribution <NUM> to a bimodal mixture of Gaussian distributions <NUM> that corresponds to the distribution <NUM>. This is also shown in <FIG>. Then, a second error may be determined between the distribution <NUM> and the bimodal mixture of Gaussian distributions <NUM>. In this example, the second error is <NUM>.

The determination whether the distribution <NUM> is unimodal or bimodal may then be based at least partially upon the first error and/or the second error. More particularly, it may be determined that the distribution <NUM> is unimodal when the first error is less than the second error, and it may be determined that the distribution <NUM> is bimodal when the second error is less than the first error. In this particular example, the second error is less because the bimodal mixture of Gaussian distributions <NUM> more closely approximates the distribution <NUM>. Thus, the distribution <NUM> is determined to be bimodal, meaning that the first image <NUM> includes two (e.g., different) distributions.

When the distribution <NUM> is unimodal, it may be determined that the image <NUM> and/or the image <NUM> includes a single distribution (e.g., all ground region <NUM> and no sky region <NUM>). The method <NUM> may then loop back to step <NUM>. When the distribution <NUM> is bimodal, it may be determined that the image <NUM> and/or the image <NUM> includes the sky region <NUM>. For example, it may be determined that the image <NUM> and/or the image <NUM> includes two distributions (e.g., a combination of the ground region <NUM> and the sky region <NUM>). The method <NUM> may then proceed as described below.

In response to determining that the distribution <NUM> is bimodal, the method <NUM> may also include dividing the image <NUM> and/or the image <NUM> to produce a second (e.g., divided) image <NUM>, as at <NUM>. This step may be performed by the computing system <NUM>. More particularly, the computing system <NUM> may divide the image <NUM> and/or the image <NUM> (e.g., into the two distributions) using adaptive binary intensity thresholding or Otsu's binarization method. The second image <NUM> is shown in <FIG>. The second image <NUM> may include a plurality of first (e.g., black) pixels <NUM> that correspond to the ground region <NUM> and a plurality of second (e.g., white) pixels <NUM> that correspond to the sky region <NUM>.

The method <NUM> may also include filtering the second image <NUM> to produce a third (e.g., filtered) image <NUM>, as at <NUM>. This step may be performed by the computing system <NUM>. This step may include converting one or more subsets (three are labelled: <NUM>-<NUM>) of the first (e.g., black) pixels <NUM> in the second image <NUM> into the second (e.g., white) pixels <NUM>. This is shown in <FIG>. A border is hatched around the subsets <NUM>-<NUM> to make them more easily identifiable. The subsets <NUM>-<NUM> represent the first (e.g., black) pixels <NUM> that are at least partially (or fully) surrounded by the second (e.g., white) pixels <NUM>. As will be appreciated, no portion of the ground region <NUM> should be in or surrounded by the sky region <NUM>, so this step is converting those portions of the ground region <NUM> into the sky region <NUM> so that there is one contiguous sky region <NUM>.

This step may also include converting one or more subsets (three are labelled: <NUM>-<NUM>) of the second (e.g., white) pixels <NUM> in the second image <NUM> into the first (e.g., black) pixels <NUM>. This is shown in <FIG>. A border is hatched around the subsets <NUM>-<NUM> to make them more easily identifiable. The subsets <NUM>-<NUM> represent the second (e.g., white) pixels <NUM> that are at least partially (or fully) surrounded by the first (e.g., black) pixels <NUM>. As will be appreciated, no portion of the sky region <NUM> should be in or surrounded by the ground region <NUM>, so this step is converting those portions of the sky region <NUM> into the ground region <NUM> so that there is one contiguous ground region <NUM>. As mentioned above, filtering the second image <NUM> may produce the third (e.g., filtered) image <NUM>, which is shown in <FIG>.

The method <NUM> may also include identifying a horizon <NUM> between the first pixels <NUM> and the second pixels <NUM> in the third image <NUM>, as at <NUM>. This step may be performed by the computing system <NUM>. The horizon <NUM> may be a substantially horizontal line; however, the horizon <NUM> may have vertical variations corresponding to mountains, valleys, buildings, trees, etc..

In an implementation, the method <NUM> may also include combining the first image <NUM> and at least a portion of the third image <NUM> to produce a fourth (e.g., combined) image <NUM>, as at <NUM>. This is shown in <FIG>. This step may be performed by the computing system <NUM>. This may include overlaying at least a portion of the third image <NUM> on the first image <NUM> to produce the fourth image <NUM>. The portion of the third image <NUM> that is overlaid substantially corresponds to the ground region <NUM> in the first image <NUM>. The fourth image <NUM> may include the horizon <NUM>. As shown, the portion of the third image <NUM> may be transparent or opaque such that the first image <NUM> underneath may be seen. In <FIG>, the portion of the third image <NUM> is hatched to be seen more clearly. In another example, the portion of the third image <NUM> may be solid such that the first image <NUM> underneath may not be seen. In yet another example, this step may be omitted.

The method <NUM> may also include navigating (e.g., steering) the aircraft <NUM>, as at <NUM>. The aircraft <NUM> may be navigated (e.g., steered) based at least partially upon the third image <NUM>, the horizon <NUM>, the fourth image <NUM>, or a combination thereof. For example, the aircraft's pitch and/or roll may be determined based at least partially upon the horizon <NUM>, and the aircraft <NUM> may be steered to adjust the pitch and/or roll. The navigation may be performed (e.g., automatically) by the computing system <NUM>. In another example, the navigation may be performed by a user. The user may be in the aircraft <NUM> (e.g., a pilot), or the user may be on the ground and steering the aircraft <NUM> remotely.

In another example, the method <NUM> may also or instead include detecting one or more objects, as at <NUM>. The objects may be detected based at least partially upon the third image <NUM>, the horizon <NUM>, the fourth image <NUM>, or a combination thereof. For example, the third image <NUM>, the horizon <NUM>, and/or the fourth image <NUM> may be used as an input into a path-planning or object detection algorithm. The objects detected may be or include moving objects. For example, the objects may be or include other aircrafts in flight, and the aircraft <NUM> may be navigated in response to (e.g., to avoid) the other aircrafts in flight. The objects may be tracked by this method <NUM> continuously.

As used herein, the terms "inner" and "outer"; "up" and "down"; "upper" and "lower"; "upward" and "downward"; "upstream" and "downstream"; "above" and "below"; "inward" and "outward"; and other like terms as used herein refer to relative positions to one another and are not intended to denote a particular direction or spatial orientation. The terms "couple," "coupled," "connect," "connection," "connected," "in connection with," and "connecting" refer to "in direct connection with" or "in connection with via one or more intermediate elements or members. " Similarly, the terms "bonded" and "bonding" refer to "directly bonded to" or "bonded to via one or more intermediate elements, members, or layers.

Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the disclosure are approximations, the numerical values set forth in the specific examples are reported as precisely as possible. Any numerical value, however, inherently contains certain errors necessarily resulting from the standard deviation found in their respective testing measurements. Moreover, all ranges disclosed herein are to be understood to encompass any and all sub-ranges subsumed therein.

While the present teachings have been illustrated with respect to one or more examples, alterations and/or modifications can be made to the illustrated examples without departing from the scope of the appended claims. In addition, while a particular feature of the present teachings may have been disclosed with respect to only one of several examples, such feature may be combined with one or more other features of the other examples as may be desired and advantageous for any given or particular function. As used herein, the terms "a", "an", and "the" may refer to one or more elements or parts of elements. As used herein, the terms "first" and "second" may refer to two different elements or parts of elements. As used herein, the term "at least one of A and B" with respect to a listing of items such as, for example, A and B, means A alone, B alone, or A and B. Those skilled in the art will recognize that these and other variations are possible. Furthermore, to the extent that the terms "including," "includes," "having," "has," "with," or variants thereof are used in either the detailed description and the claims, such terms are intended to be inclusive in a manner similar to the term "comprising. " Further, in the discussion and claims herein, the term "about" indicates that the value listed may be somewhat altered, as long as the alteration does not result in nonconformance of the process or structure to the intended purpose described herein. Finally, "exemplary" indicates the description is used as an example, rather than implying that it is an ideal.

Claim 1:
A method (<NUM>), comprising:
receiving a first image, or identifying a first image (<NUM>) from a plurality of images, wherein the first image is captured by a camera;
identifying a number of edges (<NUM>, <NUM>, <NUM>) in the first image (<NUM>); determining a distribution (<NUM>) based at least partially upon an intensity of each pixel in the first image (<NUM>) when the number of edges (<NUM>, <NUM>, <NUM>) is greater than a predetermined threshold;
determining that the distribution (<NUM>) is bimodal;
dividing the first image (<NUM>) to produce a second image (<NUM>) in response to determining that the distribution approximates a bimodal mixture of Gaussian distributions (<NUM>), wherein the second image (<NUM>) comprises a plurality of first pixels (<NUM>) corresponding to a ground region and a plurality of second pixels (<NUM>) corresponding to a sky region; and
determining that a horizon (<NUM>) is defined between the plurality of first pixels (<NUM>) and the plurality of second pixels (<NUM>).