Patent Description:
Sky segmentation uses a machine-learning model to associate pixels in an image or video frame with the sky. Sky segmentation methods are typically either edge-based or content-based. Edge-based sky segmentation methods do not produce a complete pixel-by-pixel sky segmentation. Rather, they conclude with finding a horizon line for micro air vehicle attitude estimation. Content-based sky segmentation methods require large amounts of pixel-by-pixel hand-labeled data, and they also have a high computational cost. In addition to the foregoing shortcomings, neither method works well with limited labeled data and real-time runtime constraints.

<NPL>, in accordance with its Abstract, states detecting sky-sea line is very important to identify targets for missile imaging guidance. As traditional Canny algorithm gives out too many edges in infrared images with sky-sea background, an modified Canny algorithm is suggested to detect the sky-sea line. In this algorithm, the Canny method is combined with a median filter, its two thresholds are designed to be adaptive based on the properties of these infrared images. A criterion is set up to distinguish the sky-sea line from other objects in an image.

<NPL>, in accordance with its Abstract, states skyline detection plays an important role in unmanned aerial vehicle visual flight control systems. Many studies assume that the skyline in the image is a straight line with clear edge features, but this assumption is not necessarily true under different weather and environmental conditions. For example, when it is cloudy or foggy the contrast and brightness of the image becomes weaker and the skyline is not obvious. The edge features of skyline are hard to be found in these cases. When an aircraft is flying at low altitudes, mountains, woods, or buildings become part of the skyline. Because of this, the contour of the skyline is not a straight line anymore, and the assumptions contradict the actual situation. In order to solve these problems, this paper proposes an algorithm that can detect the contour of a curving skyline in an image under different brightnesses and contrasts. Experiments are conducted using <NUM> images taken in different weather and environmental conditions. The results can be used as a reference for unmanned aerial vehicle flight in order to determine flight attitude.

<NPL> in accordance with its Abstract, states sea and sky boundary identification (i.e. marine horizon line detection) from a marine image is a problem of great interest for reasons such as, unmanned surface or aerial vehicle navigation, surveillance by object detection and tracking, and determining the spatial orientation of the ship. Due to the complexity of the marine environment, the problem poses its own unique challenges. In recent years, different methods have been proposed by the researchers to solve the problem. Those methods can be grouped into two categories; (i) edge detection based horizon detection, and (ii) machine learning-based horizon detection. In this paper, we present a survey on edge detection based recent marine horizon line detection methods and their applications. We have selected studies from the previous three years and discussed each study's approach to marine horizon line detection issue, the datasets used for testing purposes and its results. The authors' observations for each study are presented with a recommendation for their suitability for a specific application in the marine environment. Findings of the survey and future research directions for the researchers are also identified and presented.

There is provided a method of sky segmentation. The method includes receiving a first image. The method also includes detecting a plurality of edges in the first image. The method also includes connecting the edges. The method also includes identifying a largest contour in the first image based at least partially upon the connected edges. The method also includes determining a convex hull of the largest contour. The method also includes generating a second image comprising the convex hull.

A computing system is also disclosed. The computing system includes one or more processors and a memory system. The memory system includes one or more non-transitory computer-readable media storing instructions that, when executed by at least one of the one or more processors, cause the computing system to perform operations. The operations include receiving a first image. The first image is captured by a camera on an aircraft in flight. The operations also include detecting a plurality of edges in the first image based at least partially upon a threshold. The operations also include connecting the edges. The operations also include identifying a largest contour in the first image based at least partially upon the connected edges. The operations also include determining a convex hull of the largest contour. The convex hull represents a portion of a ground region in the first image. The operations also include generating a second image comprising the convex hull. A plurality of first pixels in the second image represent the convex hull, and a plurality of second pixels in the second image do not represent the convex hull.

In other examples, the operations include receiving a first image. The first image is captured by a camera on a vehicle. The operations also include detecting a plurality of edges in the first image based at least partially upon a threshold. The operations also include connecting the edges. The edges are connected using morphological closing with a square kernel, and the morphological closing uses a kernel having an area from about <NUM> pixels to about <NUM> pixels (or from <NUM> pixels to <NUM> pixels). The operations also include identifying a largest contour in the first image based at least partially upon the connected edges. The largest contour includes a largest area. The operations also include determining a convex hull of the largest contour. The convex hull represents a portion of a ground region in the first image. The operations also include generating a second image including the convex hull. A plurality of first pixels having a first color in the second image represent the convex hull, and a plurality of second pixels having a second color in the second image do not represent the convex hull. The operations also include identifying a bottom-most first pixel in each column in the second image. The operations also include converting the second pixels that are below the bottom-most first pixel in each column to first pixels to produce a modified second image. The first pixels in the modified second image represent the ground region, and the second pixels in the modified second image represent a sky region. The operations also include combining at least a portion of the first image and at least a portion of the modified second image to produce a combined image.

The edges may be connected using morphological closing, and wherein the morphological closing uses a kernel having an area from about <NUM> pixels to about <NUM> pixels or from <NUM> pixels to <NUM> pixels.

An aircraft may be steered based at least partially upon the modified second image. A second aircraft may be detected in flight based at least partially upon the modified second image.

The operations may further comprise transmitting or displaying a notification to steer the aircraft based at least partially upon the combined image, or based at least partially upon a trajectory of the second aircraft.

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the present teachings and together with the description, serve to explain the principles of the present teachings.

It should be noted that some details of the figures have been simplified and are drawn to facilitate understanding rather than to maintain strict structural accuracy, detail, and scale.

Reference will now be made in detail to the present teachings, examples of which are illustrated in the accompanying drawings. In the drawings, like reference numerals have been used throughout to designate identical elements. In the following description, reference is made to the accompanying drawings that form a part thereof, and in which is shown by way of illustration specific examples of practicing the present teachings. The following description is, therefore, merely exemplary.

The systems and methods disclosed herein may identify/produce a boundary that approximates the separation between a ground region and a sky region in an image. The systems and methods may also mask out the non-sky (e.g., ground) region from the image to produce a sky background, which may minimize false positives of the boundary and/or detected objects. The system and methods disclosed herein may require no training data and run at or near real-time to generate a pixel-by-pixel sky segmentation map. Once the boundary has been detected, the system and method may use the boundary to help detect objects (e.g., other aircrafts, birds, etc.) and to navigate the aircraft to avoid such objects. The accuracy of the detection process may be enhanced by employing the sky segmentation technique described herein.

The systems and methods may quickly and accurately detect non-sky regions within an image which have a nonlinear border. In addition, the systems and methods do not require labeled data to train a machine-learning (ML) model. The systems and methods segment non-sky regions pixel-by-pixel, rather than by finding a horizon line and dividing the image into "above horizon" and "below horizon.

The camera may be fixed to the aircraft at a defined position relative to the aircraft with its field of view directed toward a region of interest (e.g., forward and/or in the direction that the aircraft is travelling). The sky regions may be entirely contained above non-sky regions. The non-sky regions may contain more edge sections than sky regions do. The sky and non-sky regions are separated by a horizon: a moderately straight line which traverses most to all of the width of the image. Examples of the system and method may divide the image into sky and non-sky regions.

<FIG> illustrates a schematic view of an aircraft <NUM> in flight, according to an example. The aircraft <NUM> may be or include an airplane, a helicopter, an unmanned aerial vehicle (e.g., a drone), a spacecraft, or the like. The aircraft <NUM> may include a camera <NUM>. The camera <NUM> may be coupled the aircraft <NUM> and/or positioned within the aircraft <NUM>. The camera <NUM> may be configured to capture one or more images. The camera <NUM> may also or instead be configured to capture a video. In some examples, the camera <NUM> may be configured to capture a continual stream of images over time (i.e. a video), and the images may be still frames taken from the video.

The images and/or video may be transmitted to a computing system <NUM> on the aircraft <NUM>. In other examples, the computing system <NUM> may be located on the ground (e.g., in a control station) in communication with an on-board computing system that is on/in the aircraft <NUM>. The computing system <NUM> may include one or more processors and a memory system. The memory system may include one or more non-transitory computer-readable media storing instructions that, when executed by at least one of the one or more processors, cause the computing system <NUM> to perform operations. The operations are described with reference to <FIG> below.

<FIG> illustrates a flowchart of a method <NUM> for segmenting an image into a sky region and a ground region, according to an example. An illustrative order of the method <NUM> is described below. One or more steps of the method <NUM> may be performed in a different order, repeated, or omitted.

The method <NUM> may include receiving a first image, as at <NUM>. An example of the first image <NUM> is shown in <FIG>. The first image <NUM> may be received by the computing system <NUM>. As mentioned above, the first image <NUM> may be captured by the camera <NUM>. The camera <NUM> may be in an upright position and pointed forward such that the first image <NUM> includes a region of interest in the direction that the aircraft <NUM> is moving. Thus, the first image <NUM> may include at least a portion of the flight path of the aircraft <NUM>. The first image <NUM> may include a ground region <NUM>, a sky region <NUM>, or both.

The method <NUM> may also include detecting a plurality of edges (three are identified: <NUM>, <NUM>, <NUM>) in the first image <NUM>, as at <NUM>. <FIG> shows the edges <NUM>, <NUM>, <NUM> detected/identified in the first image <NUM>. As used herein, an "edge" refers to a point, set of points, or line (e.g., one or more pixels) in an image where the brightness changes sharply (e.g., by more than a predetermined threshold). Although the first image <NUM> includes more than three edges, for the sake of simplicity, only three are identified. This step may be performed by the computing system <NUM>. More particularly, the computing system <NUM> may use canny edge detection (e.g., with low hysteresis parameters) or Laplacian edge detection methods to detect the edges <NUM>, <NUM>, <NUM> in the first image <NUM>. In at least one example, the detection may be based at least partially upon a sensitivity threshold. For example, the sensitivity threshold may be increased (e.g., by a user) to detect more edges in the first image <NUM>, or the sensitivity threshold may be decreased to detect fewer edges in the first image <NUM>. In at least one example, the first image <NUM> may be binary, and pixels representing an edge <NUM>, <NUM>, <NUM> may have a value of zero, and pixels that do not represent an edge <NUM>, <NUM>, <NUM> may have a value of one, or vice versa.

The method <NUM> may also include connecting one or more of the edges <NUM>, <NUM>, <NUM> in the first image <NUM>, as at <NUM>. <FIG> shows the edges <NUM>, <NUM>, <NUM> (from <FIG>) after being connected to produce connected edges <NUM>, <NUM>, <NUM>. This step may be performed by the computing system <NUM>. More particularly, the computing system <NUM> may use morphological closing to produce the connected edges <NUM>, <NUM>, <NUM>. The morphological closing may use a kernel with an area from about <NUM> pixels to about <NUM> pixels (or <NUM> pixels to <NUM> pixels), about <NUM> pixels to about <NUM> pixels (or <NUM> pixels to <NUM> pixels), about <NUM> pixels to about <NUM> pixels (or <NUM> pixels to <NUM> pixels), or about <NUM> pixels to about <NUM> pixels (or <NUM> pixels to <NUM> pixels). The kernel may have a shape that is substantially a square, a rectangle, a triangle, a circle, an oval, an ellipse, or the like (or that is a square, a rectangle, a triangle, a circle, an oval, an ellipse, or the like).

The method <NUM> may also include identifying one or more contours (three are shown: <NUM>, <NUM>, <NUM>) in the first image <NUM>, as at <NUM>. <FIG> shows the contours <NUM>, <NUM>, <NUM> in the first image <NUM>. This step may be performed by the computing system <NUM>. The contours <NUM>, <NUM>, <NUM> may be identified based at least partially upon the connected edges. For example, the contours <NUM>, <NUM>, <NUM> may each have an uninterrupted perimeter that is defined by the connected edges. In some examples, the first image <NUM> may include a plurality of contours <NUM>, <NUM>, <NUM>, and this step may include identifying the largest contour <NUM>, which is outlined with a hatching pattern to be seen more clearly. The largest contour <NUM> may be or include the contour with the most pixels (e.g., the largest area).

The method <NUM> may also include determining a convex hull <NUM> of the (e.g., largest) contour <NUM>, as at <NUM>. <FIG> shows the convex hull <NUM> of the contour <NUM>. This step may be performed by the computing system <NUM>. The convex hull <NUM> is outlined with a hatching pattern to demonstrate and describe the step <NUM>. In practice, such hatching may or may not be used. As used herein, a "convex hull" refers to the smallest convex shape that (e.g., entirely) surrounds another shape (e.g., the contour <NUM>). Thus, no line can be drawn from a first point inside of the shape to a second point inside of the shape that at some point along the line falls outside of the shape. The convex hull <NUM> represents a portion of the ground region <NUM> in the first image <NUM>. The top <NUM> of the convex hull <NUM> may represent the horizon (e.g., between the ground region <NUM> and the sky region <NUM>).

The method <NUM> may also include generating a second image <NUM>, which includes the convex hull <NUM>, as at <NUM>. Alternatively, the first image <NUM> may be modified to produce the second image <NUM>, which includes the convex hull <NUM>. <FIG> shows the second image <NUM>, which includes the convex hull <NUM>. This step may be performed by the computing system <NUM>. The second image <NUM> may include a plurality of first (e.g., black) pixels <NUM> and a plurality of second (e.g., white) pixels <NUM>. The first pixels <NUM> may represent the convex hull <NUM>, which represents at least a part of the ground region <NUM>. The second pixels <NUM> do not represent the convex hull <NUM>. Instead, at least a portion of the second pixels <NUM> may represent the sky region <NUM>. For example, the second pixels <NUM> that are above the first pixels <NUM> may represent the sky region <NUM>. However, as may be seen, some of the second pixels <NUM> are below the first pixels <NUM>. As the sky region <NUM> cannot be below the ground region <NUM>, this is corrected below.

The method <NUM> may also include identifying a bottom-most first pixel <NUM> in a column <NUM> in the second image <NUM>, as at <NUM>. This is shown in <FIG>. This step may be performed by the computing system <NUM>. This step may be repeated for each column in the second image <NUM>. This may yield a bottom-most layer <NUM> of first pixels <NUM> in the second image <NUM>. The bottom-most layer <NUM> is outlined with a hatching pattern to be seen more clearly.

The method <NUM> may also include converting the second pixels <NUM> that are below the bottom-most first pixel <NUM> in the column <NUM> to first pixels <NUM> to produce a modified second image <NUM>, as at <NUM>. This is shown in <FIG>. This step may be performed by the computing system <NUM>. This corrects the issue noted above. Now, the first pixels <NUM> represent the ground region <NUM>, and the second pixels <NUM> represent the sky region <NUM>. There is no longer a portion of the sky region <NUM> that is positioned below the ground region <NUM>. The modified second image <NUM> may be or include a sky mask. As used herein, a "sky mask" refers to a binary image where "false" (e.g., <NUM>) values indicate ground pixels and "true" (e.g., <NUM>) values indicate sky pixels.

In some examples, the method <NUM> may also include combining the first image <NUM> and at least a portion of the modified second image <NUM> to produce a combined image <NUM>, as at <NUM>. This is shown in <FIG>. This step may be performed by the computing system <NUM>. In at least one example, this may include overlaying at least a portion of the modified second image <NUM> on the first image <NUM> to produce the combined image <NUM>. The portion of the modified second image <NUM> that is overlaid corresponds to the ground region <NUM> in the first image <NUM> (or substantially corresponds to the ground region <NUM> in the first image <NUM>). As shown, the portion of the modified second image <NUM> may be transparent or opaque such that the first image <NUM> underneath may be seen. In <FIG>, the portion of the modified second image <NUM> is hatched to be seen more clearly. In other examples, the portion of the modified second image <NUM> may be solid such that the first image <NUM> underneath may not be seen. In yet other examples, this step may be omitted.

The method <NUM> may also include navigating (e.g., steering) the aircraft <NUM>, as at <NUM>. The aircraft <NUM> may be navigated (e.g., steered) based at least partially upon the second image <NUM>, the modified second image <NUM>, the combined image <NUM>, or a combination thereof. The navigation may be performed (e.g., automatically) by the computing system <NUM>. In other examples, the navigation may be performed by a user. The user may be in the aircraft <NUM> (e.g., a pilot), or the user may be on the ground and steering the aircraft <NUM> remotely.

In other examples, the method <NUM> may also or instead include detecting one or more objects, as at <NUM>. This step may be performed by the computing system <NUM>. The objects may be detected based at least partially upon the second image <NUM>, the modified second image <NUM>, the combined image <NUM>, or a combination thereof. For example, the second image <NUM>, the modified second image <NUM>, and/or the combined image <NUM> may be used as an input into a path-planning or object detection algorithm. The object detection algorithm may more accurately detect objects and/or detect objects with fewer false positives when the image is segmented into the sky and non-sky regions, which enables detection of the objects above the horizon (e.g., in the sky region). The objects detected may be or include moving objects. For example, the objects may be or include other aircrafts in flight, and the aircraft <NUM> may be navigated in response to (e.g., to avoid) the other aircrafts in flight.

As used herein, the terms "inner" and "outer"; "up" and "down"; "upper" and "lower"; "upward" and "downward"; "upstream" and "downstream"; "above" and "below"; "inward" and "outward"; and other like terms as used herein refer to relative positions to one another and are not intended to denote a particular direction or spatial orientation. The terms "couple," "coupled," "connect," "connection," "connected," "in connection with," and "connecting" refer to "in direct connection with" or "in connection with via one or more intermediate elements or members. " Similarly, the terms "bonded" and "bonding" refer to "directly bonded to" or "bonded to via one or more intermediate elements, members, or layers.

Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the disclosure are approximations, the numerical values set forth in the specific examples are reported as precisely as possible. Any numerical value, however, inherently contains certain errors necessarily resulting from the standard deviation found in their respective testing measurements. Moreover, all ranges disclosed herein are to be understood to encompass any and all sub-ranges subsumed therein.

Claim 1:
A method of sky segmentation (<NUM>), comprising
receiving a first image (<NUM>);
detecting a plurality of edges (<NUM>, <NUM>, <NUM>) in the first image (<NUM>);
connecting the edges (<NUM>, <NUM>, <NUM>);
identifying a largest contour (<NUM>) in the first image (<NUM>) based at least partially upon the connected edges (<NUM>, <NUM>, <NUM>);
determining a convex hull (<NUM>) of the largest contour (<NUM>); and
generating a second image (<NUM>) comprising the convex hull (<NUM>).