Patent Description:
Aquaculture involves the farming of aquatic organisms, such as fish, crustaceans, or aquatic plants. In aquaculture, and in contrast to commercial fishing, freshwater and saltwater fish populations are cultivated in controlled environments. For example, the farming of fish can involve raising fish in tanks, fish ponds, or ocean enclosures.

Aquaculture farmers have found that it can be difficult to accurately estimate the weight of fish prior to harvesting. A manual process of catching and weighing a sample set of fish is often used to estimate the weight of a larger number of fish in an enclosure or pen, however such a process can be time-consuming, inaccurate, and requires substantial financial, logistical, and human resources.

Reference is made to <CIT> which presents a fish monitoring system deployed in a particular area to obtain fish images. Neural networks and machine-learning techniques may be implemented to periodically train fish monitoring systems and generate monitoring modes to capture high quality images of fish based on the conditions in the determined area. The camera systems may be configured according to the settings, e.g., positions, viewing angles, specified by the monitoring modes when conditions matching the monitoring modes are detected. Each monitoring mode may be associated with one or more fish activities, such as sleeping, eating, swimming alone, and one or more parameters, such as time, location, and fish type.

Further reference is made to <CIT> which presents a method and system to predict fish weight and biomass using a non-invasive, digital stereo-camera and computer vision system and method. The stereo camera system is immersed in a net fish pen and captures stereo images of freely moving fish from a substantially lateral perspective. The computer vision system automatically identifies and estimates specific combinations of fin to fin, body depth, and length dimensions, that are learned from the stereo images. The estimates are then used to predict weight with a high degree of accuracy. The system and method have the advantage of being more automated and highly accurate and providing greatly reduced stress levels in fish, compared with current biomass estimation techniques.

In general, innovative aspects of the subject matter described in this specification relate to fish weight estimation based on fish tracks identified in images. A fish track may refer to a path of a fish, in relation to a camera, across one or more images. For example, a single fish track may be an appearance of a single fish swimming closer to a camera in ten images. In another example, a single fish track may be an appearance of a single fish swimming into view of a camera for two images.

Images of fish within a fish enclosure may be captured by an underwater camera and the weights of fish may be estimated from the images. The weight estimated from the images may be used to estimate a representative weight, e.g., average, median, etc., for fish in the fish enclosure. The representative weight may be used to determine whether the fish are growing as expected or whether there are any problems with the fish. For example, the representative weight may be less than expected, which may indicate that the fish should be feed more or the fish should be allowed more time to grow.

A representative weight of fish in the fish enclosure may be determined by sampling a random subset of the weights estimated from images, and continuing to randomly add more of the weights to the subset until the representative weight of the subset stabilizes. For example, given three thousand estimated weights of fish, taking the average estimates of fish weights in a random order may result in a stable average after averaging one thousand seven hundred weights of fish.

However, a more efficient approach to determining a representative weight of the fish in the fish enclosure may be to prioritize sampling higher quality estimated weights. For example, a quality of each weight's estimate may be determined, and then weights may be selected for averaging in descending order of quality until the average weight of the subset stabilizes after averaging one thousand two hundred weights of fish. Accordingly, the more efficient approach for determining a representative weight may result in less computations necessary and reduced power consumption for computation. Additionally, a representative weight determined from a subset of higher quality weight estimates may be more accurate than an average weight determined from a random subset of weight estimates.

In some implementations, in response to the representative weight being less than expected, an automated fish feeder may be controlled to feed the fish more by increasing a rate that fish feed is dispensed and/or increasing a time that fish feed is dispensed. In some implementations, in response to the representative weight reaching a harvest threshold, an automated fish harvester may be controlled to begin harvesting the fish.

One innovative aspect of the subject matter described in this specification is embodied in a method as recited in claim <NUM>.

Other implementations of this and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices. A system of one or more computers can be so configured by virtue of software, firmware, hardware, or a combination of them installed on the system that in operation cause the system to perform the actions. One or more computer programs can be so configured by virtue of having instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. For instance, in some aspects identifying, by one or more processors, fish tracks shown in the images of the fish includes determining that a fish shown in a first image of the images is shown in a subsequently captured second image of the images. In certain aspects, obtaining images of fish enclosed in a fish enclosure includes instructing an underwater camera to capture the images at different depths in the fish enclosure and receiving the images from the underwater camera. In some implementations, determining, by the one or more processors, a representative weight of the fish in the fish enclosure based on weights of the fish shown in the subset of the fish tracks includes determining the weights of the fish based on apparent sizes of the fish in the images.

In certain aspects, actions include determining that the representative weight satisfies a harvesting criteria and triggering harvesting of the fish in the fish enclosure. In some aspects, determining a representative weight of the fish in the fish enclosure based on weights of the fish shown in the subset of the fish tracks includes selecting, by the one or more processors, a second subset from the fish tracks based on the quality scores, where the fish tracks selected in the second subset are in the subset of the fish tracks, determining, by the one or more processors, a second representative weight of the fish in the enclosure from weights of fish shown in the subset of the fish tracks in the second subset, and determining, by the one or more processors, that a difference between the representative weight and the second representative weight satisfies a stability criteria.

In some implementations, determining, by the one or more processors, that a difference between the representative weight and the second representative weight satisfies a stability criteria includes determining that the difference between the representative weight and the second representative weight is below a threshold weight. In certain aspects, determining, by the one or more processors, that a difference between the representative weight and the second representative weight satisfies a stability criteria includes determining that a first distribution of the weights of fish shown in the fish tracks in the subset is stable relative to a second distribution of the weights of fish shown in the fish tracks in the second subset.

In some aspects, selecting, by the one or more processors, a second subset from the fish tracks based on the quality scores includes determining a ranking the fish tracks based on the quality scores and selecting, as the second subset, a predetermined number of the fish tracks based on the ranking. In some implementations, selecting, by the one or more processors, a subset of the fish tracks based on the quality scores includes selecting both a second predetermined number of remaining fish tracks of the fish tracks based on the ranking and the fish tracks in the second subset. In certain aspects, the representative weight includes an average weight.

The details of one or more implementations are set forth in the accompanying drawings and the description, below. Other potential features and advantages of the disclosure will be apparent from the description and drawings, and from the claims.

The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit the implementations described and/or claimed in this document.

<FIG> is a diagram of an example system <NUM> for fish weight estimation based on fish tracks identified in images. An enclosure <NUM> contains livestock <NUM>. The livestock <NUM> can be aquatic creatures which swim freely within the confines of the enclosure <NUM>. In some implementations, the aquatic livestock <NUM> stored within the enclosure <NUM> can include finfish or other aquatic lifeforms. The livestock <NUM> can include for example, juvenile fish, koi fish, sharks, salmon, and bass, to name a few examples. Where the livestock <NUM> is fish, the enclosure <NUM> may be referred to as a fish enclosure.

In addition to the aquatic livestock, the enclosure <NUM> contains water, e.g., seawater, freshwater, or rainwater, although the enclosure can contain any fluid that is capable of sustaining a habitable environment for the aquatic livestock. In some implementations, the system <NUM> is anchored to a structure such as a pier, dock, or buoy instead of being confined within the enclosure <NUM>. For example, instead of being confined within the enclosure <NUM>, the livestock <NUM> can be free to roam a body of water, and the system <NUM> can monitor livestock within a certain area of the body of water.

The system <NUM> includes a sensor subsystem <NUM> that generates sensor data, a weight subsystem <NUM> that determines an average weight for the livestock <NUM> based on the sensor data, and a winch subsystem <NUM> that moves the sensor subsystem <NUM>. The sensor subsystem <NUM> includes a camera which can be fully submerged in the enclosure <NUM>. The position of the sensor subsystem <NUM> within the enclosure <NUM> is determined by instructions generated by the weight subsystem <NUM>.

The sensor subsystem <NUM> may be waterproof and can withstand the effects of external forces, such as typical ocean currents, without breaking. The system <NUM> can additionally store the sensor data captured by the sensor subsystem <NUM> in a sensor data storage. In some implementations, the system <NUM> can store media, such as video and images, as well as sensor data, such as ultrasound data, thermal data, and pressure data, to name a few examples. Additionally, the sensor data can include GPS information corresponding to a geolocation at which the sensor subsystem captured the sensor data.

The winch subsystem <NUM> receives the instructions and activates one or more motors to move the sensor subsystem <NUM> to the position corresponding to the instructions. The winch subsystem <NUM> can include one or more motors, one or more power supplies, and one or more pulleys to which the cord, which suspends the sensor subsystem <NUM>, is attached. Although the winch subsystem <NUM> includes a single cord, any configuration of one or more cords and one or more pulleys that allows the sensor subsystem <NUM> to move and rotate, as described herein, can be used.

The winch subsystem <NUM> receives an instruction from the weight subsystem <NUM> and activates the one or more motors to move the cord. The cord, and the attached sensor subsystem <NUM>, can be moved along the x, y, and z-directions, to a position corresponding to the instruction.

A motor of the winch subsystem <NUM> can be used to rotate the sensor subsystem <NUM> to adjust the horizontal angle and the vertical angle of the sensor subsystem. A power supply can power the individual components of the winch subsystem. The power supply can provide AC and DC power to each of the components at varying voltage and current levels. In some implementations, the winch subsystem can include multiple winches or multiple motors to allow motion in the x, y, and z-directions.

One or both of the sensor subsystem <NUM> and the winch subsystem <NUM> can include inertial measurement devices for tracking motion and determining position of the sensor subsystem <NUM>, such as accelerometers, gyroscopes, and magnetometers. The winch subsystem <NUM> can also keep track of the amount of cord that has been spooled out and reeled in, to provide another input for estimating the position of the sensor subsystem <NUM>. In some implementations the winch subsystem <NUM> can also provide torques applied to the cord, to provide input on the position and status of the sensor subsystem <NUM>. In some implementations, the sensor subsystem <NUM> can be attached to an autonomous underwater vehicle (AUV), e.g., a tethered AUV.

The weight subsystem <NUM> determines a representative weight of the livestock <NUM> based on sensor data from the sensor subsystem <NUM>. For example, the sensor subsystem <NUM> determines an average weight of <NUM> kilograms (KG) per fish based on five hundred images of fish <NUM> captured by the sensor subsystem <NUM>.

The weight subsystem <NUM> determines the representative weight by identifying fish tracks 130A-F shown in the images of the fish. For example, the weight subsystem <NUM> may identify a first fish track, Fish Track C 130C, by recognizing that a single fish swam closer to a camera in five images consecutively captured by the sensor subsystem <NUM>. In another example, the weight subsystem <NUM> may identify a second fish track, Fish Track A 130A, by recognizing that a single fish swam quickly from a left edge of a view of a camera of the sensor subsystem <NUM> to a right edge of the view of the camera of the sensor subsystem <NUM> in two consecutively captured images.

The weight subsystem <NUM> may rank the fish tracks that were identified based on quality scores. For example, the weight subsystem <NUM> may determine a quality score of <NUM>% for Fish Track C, a quality score of <NUM>% for Fish Track A, and quality score of <NUM>% for Fish Track D, and rank the fish tracks in descending order of quality from top to bottom so that Fish Track C is above Fish Track A, and Fish Track A is above Fish Track D. Various ways of generating a quality score are discussed more fully below.

The weight subsystem <NUM> may determine representative weights of fish shown in expanding subsets of the fish tracks until the representative weight stabilizes. For example, the weight subsystem <NUM> may determine that an average weight of <NUM> from the top ranked four hundred fish tracks is stable.

In some implementations, the weight subsystem <NUM> may learn a pattern for the sensor subsystem <NUM> to capture sensor data based on the subset of fish tracks when the representative weight stabilizes. For example, the weight subsystem <NUM> may learn where the sensor subsystem <NUM> was positioned fish enclosure <NUM> for the top ranked fish tracks, and then instruct the sensor subsystem <NUM> to capture more images at those positions and less images at other positions. In another example, the weight subsystem <NUM> may learn times of day when the sensor subsystem <NUM> captured images of the top ranked fish tracks, and then instruct the sensor subsystem <NUM> to capture more images at those times of day. In yet another example, the weight subsystem <NUM> may learn that fewer images are needed determine a representative weight and, in response, instruct the sensor subsystem <NUM> to capture fewer images.

The weight subsystem <NUM> may output the representative weight for display or storage at a device. For example, the weight subsystem <NUM> may output the average weight of <NUM> to a computer being used by a person. In another example, the weight subsystem <NUM> may output the average weight of <NUM> to a harvesting device that determines whether to harvest some or all of the fish based on the average weight.

In some implementations, the weight subsystem <NUM> may output the representative weight to an automated fish harvester that may determine that the representative weight has reached a harvest threshold and, in response, may begin harvesting the fish. In some implementations, the weight subsystem <NUM> may output the representative weight to an automated fish feeder that may determine the representative weight is less than expected and, in response, may increase a rate that fish feed is dispensed and/or increase a time that fish feed is dispensed. Additionally or alternatively, the weight subsystem <NUM> or the device the representative weight is output to may estimate growth rates over time to project harvest times, detect disease based on lack of growth or shape of distribution of weights, or estimate a condition factor of fish to assess fish health.

<FIG> and <FIG> are diagrams of an example weight subsystem <NUM>. The weight subsystem <NUM> may be the weight subsystem <NUM> shown in <FIG>. The weight subsystem <NUM> includes a fish track identifier <NUM> that identifies fish tracks shown in images of fish, a quality score engine <NUM> that determines quality scores for the fish tracks, a fish track subset selector <NUM> that selects a subset of the fish tracks based on the quality scores, and an weight engine <NUM> that determines a representative weight based on the subset of the fish tracks.

The fish track identifier <NUM> identifies fish tracks shown in images of fish. For example, the fish track identifier <NUM> may receive five hundred images of fish and identify eight hundred fish tracks. Each fish track may correspond to an appearance of a fish in one or more images. For example, a first image may show five fish so there may be five fish tracks that correspond to the first image. In the example, a second image captured immediately after the single image may show three of the five fish in slightly different locations, one new fish, and not show the remaining two of the five fish. Accordingly, in the example, the first and second images may together show six fish tracks, where three fish tracks are show in both images, two fish tracks are shown in the first image and not in the second image, and one fish track is shown in the second image and not in the first image.

The quality score engine <NUM> may receive indications of the fish tracks identified by the fish track identifier <NUM> and determine a quality score for each fish track. For example, the quality score engine <NUM> may determine a quality score of <NUM>%, <NUM>%, and <NUM>% for a respective first, second, and third fish track. Generally, the quality score engine <NUM> may determine quality scores based on a confidence that the fish track identifier <NUM> was actually able to track the same specific fish across multiple images, given that, among other things, there may be multiple fish, multiple poses of the same fish, and difficult lighting conditions in the images that may affect the ability to track a specific fish with high confidence across the images.

While the quality score engine <NUM> is shown separate from the fish track identifier <NUM>, in some implementations, the fish track identifier <NUM> may perform the functionality described for the quality score determinator and instead determine quality scores for fish tracks and provide the scores and estimated weights from the fish tracks to the fish track subset selector <NUM>.

The quality score engine <NUM> may determine quality scores based on one or more of a minimum camera distance of the fish tracks, a maximum camera distance of the fish tracks, an image length of the fish tracks, a stereo consistency of the fish tracks, a detection confidence of the fish tracks, or a depth variation of the fish tracks. For example, the quality score engine <NUM> may determine a quality score for each fish track based on one, two, all of the factors, or some other combination. The quality score engine <NUM> may additionally or alternatively consider different factors.

Minimum camera distance may refer to a closest distance a fish is from the sensor subsystem <NUM> as shown in any image of a fish track. For example, where a fish appears at ranges of three to ten feet away in images of a fish track, the minimum camera distance may be three feet. In another example, where a fish appears at ranges of four to seven feet away in images of a fish track, the minimum camera distance may be four feet. Fish tracks with lower minimum camera distances may correspond to higher quality scores as visual weight estimates may be assumed to be more accurate as more details of the fish may be visible.

Maximum camera distance may refer to a farthest distance a fish is from the sensor subsystem <NUM> as shown in any image of a fish track. For example, where a fish appears at ranges of three to ten feet away in images of a fish track, the maximum camera distance may be ten feet. In another example, where a fish appears at ranges of four to seven feet away in images of a fish track, the maximum camera distance may be seven feet. Fish tracks with higher maximum camera distances may correspond to higher quality scores as visual weight estimates may be assumed to be less accurate as details of the fish may be visible.

Image length may refer to a number of images that a particular fish track is shown. For example, where a fish appears in five consecutive images, the image length may be five. In another example, where a fish appears in two consecutive images, the image length may be two. Fish tracks with greater image length may correspond to higher quality scores as visual weight estimates may be assumed to be more accurate as the estimate are made from more images.

Stereo consistency may refer to difference in depth estimate between a left camera and a right camera. For example, where a left camera and right camera show a distance of an eye of a fish is five and six feet away, respectively, the stereo consistency may be one foot. In another example, where a left camera and right camera show a distance of an eye of a fish is four and four and a half feet away, respectively, the stereo consistency may be half a foot. Fish tracks with higher stereo consistency image length may correspond to lower quality scores as differences may reflect an error in recognition of features of the fish.

Detection confidence of the fish tracks may refer to a confidence that a fish track was correctly detected. For example, the detection confidence score may be <NUM>% where the fish track identifier <NUM> is unsure whether the fish shown in two consecutive images is the same fish. In another example, the detection confidence score may be <NUM>% where the fish track identifier <NUM> is fairly certain that the fish shown in two consecutive images is the same fish.

Additionally or alternatively, detection confidence of the fish tracks may refer to a confidence that a fish or pose of a fish was correctly detected. For example, the detection confidence score may be <NUM>% where the fish track identifier <NUM> is unsure whether a fish is actually shown or that a fish is actually facing a camera. In another example, the detection confidence score may be <NUM>% where the fish track identifier <NUM> is fairly certain that a fish is shown or that the face is facing a camera. Fish tracks with greater detection confidence may correspond to higher quality scores as the weight estimates may be assumed to be more accurate.

Depth variation of the fish tracks may refer to a standard deviation distance from a particular depth, e.g., an average depth at which the fish tracks are identified. For example, depth variation for a fish track may be greater for a fish track identified near a top or a bottom of the fish enclosure <NUM>, and may be less for a fish track identified near a center of a fish enclosure <NUM>. Higher depth variation in selected fish tracks may be desirable as it may correspond to a higher likelihood that varying sizes of fish are used for determining the representative weight.

The quality score engine <NUM> may determine, for each of the fish tracks, one or more of the minimum camera distance of the fish tracks, the maximum camera distance of the fish tracks, the image length of the fish tracks, the stereo consistency of the fish tracks, the detection confidence of the fish tracks, or the depth variation of the fish tracks. For example, the quality score engine <NUM> may determine an image length of five for a particular fish track based on determining that the fish track is shown in five images.

The fish track subset selector <NUM> may select a subset of the fish tracks based on the quality scores. For example, the fish track subset selector <NUM> may select the four hundred fish tracks with the highest quality scores. The fish track subset selector <NUM> may select an initial number of fish tracks, and then continually select more fish tracks until the representative weight of the selected fish tracks satisfies a stability criteria. For example, the fish track subset selector <NUM> may initially select three hundred fish tracks, and then continually select fifty more fish tracks until the average of the selected fish tracks satisfies a stability criteria.

In a more detailed example, the fish track subset selector <NUM> may select the top three hundred fish tracks and arrive at an average of <NUM>, then select the top three hundred fifty fish tracks and arrive at <NUM>, and then select top four hundred fish tracks and arrive at <NUM>, which satisfies a stability criteria.

In some implementations, the stability criteria may be that a difference between a representative weight from an earlier subset and a current subset is less than a threshold. For example, the threshold may be five grams, ten grams, twenty grams, or some other amount. Additionally or alternatively, the stability criteria may be that shapes of distributions of the weights are similar. For example, a Kolmogorov Smirnoff statistic or Kullback-Leibler Divergence may be determined for an earlier subset and a current subset and then compared to a threshold to determine whether the stability criteria is satisfied. In some implementations, the threshold and/or the number of fish tracks added to the subset each iteration may be tunable. For example, the threshold and number of fish tracks added may be increased when more accuracy is desired and the threshold and number of fish tracks added may be reduced when less accuracy is desired.

The weight engine <NUM> may determine a representative weight from the subset of fish tracks. For example, the weight engine <NUM> may determine an average weight of <NUM>. The weight engine <NUM> may determine the representative weight from the weights of the fish shown in the fish tracks in the subset. For example, the weight engine <NUM> may take the average of the four hundred weights for the four hundred fish tracks in the subset.

In some implementations, the weight engine <NUM> may determine a single estimated weight for each of the fish tracks in the subset and not determine estimated weights for fish tracks that have not yet been selected. For example, the weight engine <NUM> may generate a single 3D model of a fish from an appearance of the fish in five images of a fish track, and then estimate a weight from the 3D model. Estimating a weight for a fish track may be computationally expensive and, accordingly, the weight engine <NUM> may avoid estimating a weight for all the fish tracks by not estimating a weight for fish tracks that were not selected.

As described above, the weight engine <NUM> may determine the weight by determining that the representative weight of a current subset has stabilized relative to a prior subset. For example, as shown in <FIG>, the average weight engine <NUM> may determine that a difference of <NUM> between an average weight of <NUM> for a current subset of the top four hundred fish tracks and an average weight of <NUM> for an earlier subset of the top three hundred fifty fish tracks satisfies a stability criteria of less than five grams.

In another example, as shown in <FIG>, the weight engine <NUM> may determine that a difference of <NUM> between an average weight of <NUM> for a current subset of the top four hundred fish tracks and an average weight of <NUM> for an earlier subset of the top three hundred fifty fish tracks does not satisfy a stability criteria of less than five grams and, in response, then select the top four hundred fifty fish tracks and determine that a difference of <NUM> between an average weight of <NUM> for the top four fifty hundred fish tracks and an average weight of <NUM> for the top four hundred fish tracks satisfies a stability criteria of less than five grams.

<FIG> is a flow diagram for an example process <NUM> of fish weight estimation based on fish tracks identified in images. The example process <NUM> may be performed by various systems, including the weight subsystem <NUM> shown in <FIG> or the weight subsystem <NUM> shown in <FIG>.

Briefly, and as will be described further below, the process <NUM> includes obtaining images of fish enclosed in a fish enclosure (<NUM>), identifying fish tracks shown in the images of the fish (<NUM>), determining a quality score for each of the fish tracks (<NUM>), selecting a subset of the fish tracks based on the quality scores (<NUM>), determining a representative weight of the fish shown in the subset of the fish tracks (<NUM>), and outputting the representative weight (<NUM>).

The process <NUM> includes obtaining images of fish enclosed in a fish enclosure (<NUM>). For example, the fish track identifier <NUM> may receive two hundred images of the fish. In some implementations, obtaining images of fish enclosed in a fish enclosure includes instructing an underwater camera to capture the images at different depths in the fish enclosure and receiving the images from the underwater camera. For example, the weight subsystem <NUM> may instruct the sensor subsystem <NUM> to capture an image every second and instruct the winch subsystem <NUM> to slowly move the sensor subsystem <NUM> from a top of the fish enclosure <NUM> to the bottom, and then receive the images of the fish captured by the sensor subsystem <NUM>. Fish that are different weights may prefer different depths. For example, smaller, lighter fish may prefer more shallow water than larger, heavier fish. Accordingly, moving the camera through different depths may provide images that better represent varying sizes of fish.

The process <NUM> includes identifying fish tracks shown in the images of the fish (<NUM>). For example, the fish track identifier <NUM> may identify a first fish track, Fish Track C, by recognizing that a single fish swam closer to a camera in five images consecutively captured by the sensor subsystem <NUM>. In another example, the weight subsystem <NUM> may identify a second fish track, Fish Track A, by recognizing that a single fish quickly swam from a left edge of a view of a camera of the sensor subsystem <NUM> to a right edge of the view of the camera of the sensor subsystem <NUM> in two consecutively captured images.

In some implementations, identifying fish tracks shown in the images of the fish includes determining that a fish shown in a first image of the images is shown in a subsequently captured second image of the images. For example, the fish track identifier <NUM> may determine that a fish shown in the first image is the same fish shown in a second image based on one or more of recognizing white spots on the fish, a location of the fish, a pose of the fish, or a direction of movement of the fish, etc..

The process <NUM> includes determining a quality score for each of the fish tracks (<NUM>). For example, the quality score engine <NUM> may determine a quality score of <NUM>% for Fish Track C, a quality score of <NUM>% for Fish Track A, and quality score of <NUM>% for Fish Track D. In some implementations, determining a quality score for each of the fish tracks is based on at least one of a minimum camera distance of the fish tracks, a maximum camera distance of the fish tracks, image length of the fish tracks, stereo consistency of the fish tracks, detection confidence of the fish tracks, or depth variation of the fish tracks.

For example, the quality score engine <NUM> may determine that for Fish Track C a minimum camera distance was one foot, a maximum camera distance was ten feet, an image length was five images, a stereo consistency indicates high matching, detection confidence is high, and a depth variation is high. In another example, the quality score engine <NUM> may determine that for Fish Track D a minimum camera distance was fifteen feet, a maximum camera distance was twenty feet, an image length was two images, a stereo consistency indicates low matching, detection confidence is low, and a depth variation is low.

The process <NUM> includes selecting a subset of the fish tracks based on the quality scores (<NUM>). For example, the fish track subset selector <NUM> may select a top ranked four hundred fish tracks. In some implementations, selecting a second subset from the fish tracks based on the quality scores includes determining a ranking of the fish tracks based on the quality scores and selecting, as the second subset, a predetermined number of the fish tracks based on the ranking. For example, the fish track subset selector <NUM> may rank two thousand fish tracks in descending order based on quality score and then select three hundred fish tracks from the top.

In some implementations, selecting a subset of the fish tracks based on the quality scores includes selecting both a second predetermined number of remaining fish tracks of the fish tracks based on the ranking and the fish tracks in the second subset. For example, the fish track subset selector <NUM> may continually select fifty more of the next top fish tracks to add to the subset until the representative of the subset stabilizes.

The process <NUM> includes determining a representative weight of the fish shown in the subset of the fish tracks (<NUM>). For example, the weight engine <NUM> may determine an average weight of <NUM>. In some implementations, determining a representative weight of the fish in the fish enclosure based on weights of the fish shown in the subset of the fish tracks includes determining the weights of the fish based on apparent sizes of the fish in the images. For example, the weight engine <NUM> may determine the average weight of <NUM> based on taking the average of weights visually estimated for fish as shown in the images of fish.

In some implementations, determining a representative weight of the fish in the fish enclosure based on weights of the fish shown in the subset of the fish tracks includes selecting a second subset from the fish tracks based on the quality scores, where the fish tracks selected in the second subset are in the subset of the fish tracks, determining a second representative weight of the fish in the enclosure from weights of fish shown in the subset of the fish tracks in the second subset, and determining that a difference between the representative weight and the second representative weight satisfies a stability criteria. For example, the fish track subset selector <NUM> may select the top three hundred fifty top fish tracks, determine an average weight of <NUM>, and determine that a difference of <NUM> grams, between the average weight of <NUM> and the average weight of <NUM> from the top four hundred fish tracks, satisfies a stability criteria of less than five grams.

In some implementations, determining that a difference between the representative weight and the second representative weight satisfies a stability criteria includes determining that the difference between the representative weight and the second representative weight is below a threshold weight. For example, the stability criteria may be five, ten, twenty, or some other amount of grams that the difference should be less than.

In some implementations, determining that a difference between the representative weight and the second representative weight satisfies a stability criteria includes determining that a first distribution of the weights of fish shown in the fish tracks in the subset is stable relative to a second distribution of the weights of fish shown in the fish tracks in the second subset. For example, the weight engine <NUM> may determine whether the Kolmogorov Smirnoff statistic between weight distributions in the top three hundred fifty fish tracks and the top four hundred fish tracks is less than a threshold.

The process <NUM> includes outputting the representative weight (<NUM>). For example, the weight engine <NUM> may output the average weight of <NUM> to a computer being used by a person for display to the person. In another example, the weight subsystem <NUM> may output the average weight of <NUM> to a harvesting device that determines whether to harvest the fish based on the average weight.

In some implementations, the process <NUM> includes determining that the representative weight satisfies a harvesting criteria and triggering harvesting of the fish in the fish enclosure. For example, a harvesting device may determine that an average weight of <NUM> satisfies a threshold of <NUM> and, in response and without human intervention, turn on a harvesting device that directs the fish into another shallower pen where the fish may then be harvested.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the claims.

Embodiments of the invention and all of the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the invention can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter affecting a machine-readable propagated signal, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus.

Moreover, a computer can be embedded in another device, e.g., a tablet computer, a mobile telephone, a personal digital assistant (PDA), a mobile audio player, a Global Positioning System (GPS) receiver, to name just a few.

To provide for interaction with a user, embodiments of the invention can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.

Embodiments of the invention can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the invention, or any combination of one or more such back end, middleware, or front end components.

While this specification contains many specifics, these should not be construed as limitations on the scope of the claims, but rather as descriptions of features specific to particular embodiments of the invention.

Claim 1:
A computer-implemented method comprising:
obtaining images of fish (<NUM>) enclosed in a fish enclosure (<NUM>);
identifying, by one or more processors, fish tracks (130A-F) shown in the images of the fish;
determining, by the one or more processors, a quality score for each of the fish tracks;
selecting, by the one or more processors, a subset of the fish tracks based on the quality scores;
determining, by the one or more processors, a representative weight of the fish in the fish enclosure based on weights of the fish shown in the subset of the fish tracks; and
outputting the representative weight for display or storage at a device connected to the one or more processors.