Patent Description:
Aquaculture, the farming of fish, crustaceans, mollusks, aquatic plants, algae, and other organisms, is the fastest growing food sector, and it provides most fish for human consumption. However, there is a rising demand for seafood due to human population growth, increasing disposable income in the developing world (which coincides with an increase of a meat-based diet), increasing coastal populations, and general health awareness (which tends to motivate consumers to fish-based protein). Wild fish resources are already at their limits, and the world market is increasingly focusing on being more sustainable and environmentally responsible, meaning increased harvesting of wild fish is not feasible. <CIT> discusses a system and method for measuring growth of animals, such as cattle (beef and dairy), sheep, goats, and swine and, in particular, a system and method for measuring characteristics of an animal's skeletal structure using optical and acoustic devices to determine the growth potential of the animal. <CIT> discusses methods, systems, and apparatus for identification and re-identification of fish. <CIT> discusses, methods and systems for recognizing, vaccinating, sorting and documenting alive fish. The system includes a control unit, means for acquiring information about fish, transportation means to transport fish past means for acquiring information about fish, and means for vaccinating the individual fish. The system is arranged for detecting/recognizing one or more of the following parameters of the individual fish: I. length of the fish (fork length), II. height of the fish, III. Thickness of the fish, and based on this information: instruct means for sorting fish to sort out unwanted fish prior to vaccination, or instruct means for vaccinating fish, and instructing means for sorting fish after vaccination or sorting out untreated fish.

Given the infeasibility of meeting the world demand through harvesting wild fish, aquaculture represents a natural solution. However, increasing the use of aquaculture raises additional concerns such as increased disease, growth and feeding inefficiencies, and waste management. For example, increasing the number of fish in a fish farm, without mitigating measures, increases the prevalence of diseases as the proximity of fish to each other increases. The growth and feed inefficiencies are increased as more fish in proximity to each other make health and feed management more difficult, and more fish lead to more fish waste which increases the environmental impact of a fish farm on its immediate area.

Methods and systems are disclosed herein for improvements in aquaculture that allow for increasing the number and growth efficiency of fish in an aquaculture setting while still mitigating the problems above. Specifically, using a unique sorting system, fish with superior traits are identified and sorted. In conventional systems, fish may be sorted or graded based on size, or other visually evident information, for example.

In contrast to the conventional approaches, methods and systems are described herein for non-invasive procedures for identifying characteristics in fish, and sorting the fish based on the characteristics. The present systems and methods may be configured to identify and sort based on internal conditions in juvenile or other fish such as gender, growth potential, diseases resistance, etc. It should be noted that while the discussion herein focuses on fish, the described systems and methods can be similarly applied with other animals such as poultry animals, crustaceans, swine, or other animals.

According to the invention there is provided a system for sorting animals as set out in claim <NUM>.

In some embodiments: the conveyor comprises a plurality of compartments configured to receive animals and move the animals along the path; and the system comprises a camera configured to obtain a visual image of the animal in a compartment on the conveyor as the animal moves past the camera. In some embodiments, the ultrasound transducer is configured to obtain the ultrasound image of the animal in the compartment on the conveyor. The ultrasound transducer may be configured to obtain the ultrasound image while the ultrasound transducer moves along a portion of the path with the animal. The control circuitry may be configured to: determine, based on the visual image, a starting point on the animal for the ultrasound transducer, and control the ultrasound transducer to move along the portion of the path based on the starting point to obtain the ultrasound image.

In some embodiments, the control circuitry is configured to determine the characteristic of the animal based on the ultrasound image by inputting the ultrasound image to an artificial neural network, which is trained to output the characteristic based on the ultrasound image.

In some embodiments, the artificial neural network is trained to identify one or more phenotype characteristics of the animal based on the ultrasound image, and determine presence of a biomarker in the animal indicative of the characteristic output by the artificial neural network based on the one or more phenotype characteristics.

In some embodiments, the characteristic is gender of the animal, presence of disease in the animal, size of the animal, early maturation of the animal, mature parr, presence of bacterial kidney disease in the animal, heart and/or skeletal muscle inflammation in the animal, or a fat percentage of the animal.

In some embodiments, the control circuitry is configured to determine a starting point on the animal for the ultrasound image by providing a visual image of the animal to a machine vision algorithm, which is trained to determine the starting point based on the visual image.

In some embodiments, the ultrasound transducer is configured to move in at least two dimensions. The at least two dimensions comprise a first dimension along the path; and a second dimension along a body of the animal. The second dimension may be substantially perpendicular to the first dimension and the path. The ultrasound transducer may be configured to move in the first dimension and the second dimension substantially simultaneously while obtaining the ultrasonic image, starting from a starting point. In some embodiments, a width of the ultrasonic transducer and the movement in the second dimension defines an image area on the body of the animal, which includes target anatomy of the animal.

In some embodiments, the system further comprises a plurality ultrasound transducers, each controlled by the control circuitry to obtain ultrasound images of a plurality of animals in a plurality of compartments on the conveyor at the same time.

In some embodiments, the animal is a fish and a starting point for the ultrasound image corresponds to a start of an operculum of the fish.

In some embodiments, the camera is configured to obtain a red green blue (RGB) image set that includes a visual image. The ultrasound transducer is configured to obtain an ultrasound image set of the animal that includes the ultrasound image; and the control circuitry is configured to: determine a starting point for the ultrasound transducer based on the RGB image set, and determine the characteristic based on the ultrasound image set.

In some embodiments, the animal is a fish and the visual image is a red green blue (RGB) image. In some embodiments, the control circuitry is further configured for determining characteristics based on RGB images and/or other information. For example, in some embodiments, the control circuitry is further configured to determine, based on the RGB image, a short operculum in the fish and/or damage to gills of the fish, diseases resistance, growth performance, current diseases of the fish, a smoltification status, and/or other characteristics (e.g., see other examples of characteristics described herein).

According to the invention there is further provided a method for sorting animals as set out in claim <NUM>.

The efficiencies and performance of animals such as fish are thus increased without the drawbacks discussed above.

Various other aspects, features, and advantages of the invention will be apparent through the detailed description of the invention and the drawings attached hereto. It is also to be understood that both the foregoing general description and the following detailed description are examples and not restrictive of the scope of the invention. As used in the specification and in the claims, the singular forms of "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. In addition, as used in the specification and the claims, the term "or" means "and/or" unless the context clearly dictates otherwise.

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the invention. It will be appreciated, however, by those having skill in the art that the embodiments of the invention may be practiced without these specific details or with an equivalent arrangement. In other cases, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the embodiments of the invention.

In contrast to conventional approaches to identifying animal characteristics that use invasive approaches, methods and systems are described herein for non-invasive procedures that identify animal characteristics non-invasively and efficiently, and sort the animals based on the characteristics, for farming or other applications. A sufficiently high throughput (animals per hour) in a system with a limited size (to be able to transport and potentially share between farms, plus producers have limited space) is provided. Advantageously, even with the high throughput, the system produces ultrasound images which are readable by a machine learning model (such as a neural network) so that the machine learning model can produce predictions with a very high accuracy on features like gender determination, for example, in real time or near real time.

Sufficiently high throughput is achieved by, among other things, configuring the system such that the animals are positioned side by side (instead of nose to tail) on a conveyor, which reduces the time between animals. Ultrasound can be limited in its image acquisition speed. Therefore the relative speed of a scan (how fast an ultrasound transducer scans an animal) is limited. In order to overcome these or other limitations related to scanning speed, the ultrasound transducer(s) of the present system is (are) configured to travel with an animal along the conveyor or path during a scan. This provides a faster processing time of animals by the system relative to prior approaches. In addition, a high frequency ultrasound transducer is used, but the speed of a given scan is limited to produce blur free images.

In some embodiments, ultrasound with increased acquisition speed is used so that ultrasound transducers of the present systems and methods need not travel with the animal along a conveyor or path during a scan.

The system also uses machine vision to determine the starting point of the ultrasound scan based on red green blue (RGB) images of an animal. Based on the width of an ultrasound transducer (e.g., that corresponds to a width on the animal), a certain window (on the animal) is scanned (e.g., over a certain length of the body of the animal), which may be used to predict an organ (gonad), or other characteristics of the animal.

<FIG> illustrates a perspective view of a sorting system <NUM>, in accordance with one or more embodiments. Sorting system <NUM> is configured for sorting fish <NUM> in this example, but may be configured similarly for sorting other animals. Examples of other animals include poultry animals (e.g., chickens, turkeys, etc.), crustaceans, swine, or other animals. As show in <FIG>, system <NUM> includes a conveyor <NUM>, a camera <NUM> (shown inside a camera housing <NUM> in <FIG>), ultrasound transducers 106a, 106b, and 106c, a sorter <NUM>, and/or other components.

Conveyor <NUM> is configured to receive animals (e.g., such as fish <NUM>) and move the animals along a path <NUM>. The animals may be placed one by one on conveyor <NUM>, aligned laterally (relative to an axis of conveyor <NUM>) in compartments <NUM> so that the animals travel with one of their sides facing toward camera <NUM> or an ultrasound transducer 106a, 106b, or 106c. The animals move on conveyor <NUM> to camera <NUM> and ultrasound transducers 106a, 106b, or 106c, where they are imaged (visually and ultrasonically respectively). Once the visual and ultrasound images are processed, the animals are sorted into (e.g., three or more) groups by sorter <NUM>.

Path <NUM> is aligned along conveyor <NUM>, starting at one end of conveyor <NUM> and extending to an opposite end of conveyor <NUM> and sorter <NUM>. In some embodiments, path <NUM> begins at a feeder input zone, where an operator places one animal after another, oriented in a specific direction, into compartments <NUM> of conveyor <NUM>. In some embodiments, conveyor <NUM> is configured with an axial tilt angle <NUM> so that the animals travel aligned to one side of conveyor <NUM> (e.g., caused by gravity). For example, in some embodiments, conveyor <NUM> comprises a plurality of compartments <NUM> configured to receive and hold the animals while they move along path <NUM>. In some embodiments, compartments <NUM> are oriented at an angle relative to a surface of conveyor <NUM> to ensure a repeating position of an animal in each compartment <NUM>. A given compartment <NUM> may have one or more sides that extend a certain distance from a surface of conveyor <NUM> at a certain angle. The distance or angle may be determined based on the type or dimensions of the animal, or other information. The distance or angle may be configured to be sufficient to separate one animal from the next on conveyor <NUM>.

In some embodiments, conveyor <NUM> or the surfaces of compartments <NUM> may be formed by or coated with a material configured to reduce slippage or other movement of an animal in a compartment <NUM> on conveyor <NUM>. For example, the material may include cloth, sponge, rubber or another polymer, or other materials. However, in some embodiments, one or more surfaces of conveyor <NUM> or compartments <NUM> may be metallic or be formed from other materials. In some embodiments conveyor <NUM> is supported by a frame <NUM> and/or other components.

By way of a non-limiting example, in some embodiments, the animals may be fish <NUM>. Compartments <NUM> may be configured to hold a given fish perpendicular to path <NUM> of conveyor <NUM> (shown in <FIG>). Camera <NUM> is configured to obtain visual images of animals in compartments <NUM> on conveyor <NUM> as the animals move past camera <NUM>. In some embodiments, camera <NUM> is configured to obtain a red green blue (RGB) image set that includes the visual images. For example, in some embodiments, the visual images may include an image set of an animal. The image set may include one or more images of the animal. If the image set includes multiple images, the multiple images may be captured from different angles (e.g., a top view, side view, bottom view, etc.) and/or may be captured substantially simultaneously. The views may also include plan, elevation, and/or section views. The one or more views may create a standardized series of orthographic two-dimensional images that represent the form of the three-dimensional animal. For example, six views of the animal may be used, with each projection plane parallel to one of the coordinate axes of the animal. The views may be positioned relative to each other according to either a first-angle projection scheme or a third-angle projection scheme. The images in the image set may include separate images (e.g., images stored separately, but linked by a common identifier such as a serial number) or images stored together. An image in an image set may also be a composite image (e.g., an image created by cutting, cropping, rearranging, and/or overlapping two or more images).

In some embodiments, the image set may be created using a camera <NUM> that detects electromagnetic radiation with wavelengths between about <NUM> nanometers to about <NUM> nanometers. In some embodiments, the image set may be created using a camera <NUM> that detects electromagnetic radiation with wavelengths between <NUM> to <NUM> nanometers, between <NUM> to <NUM> nanometers, between <NUM> to <NUM> nanometers, or between <NUM> to <NUM> nanometers.

Camera housing <NUM> is configured to define an imaging area for camera <NUM>. The imaging area may be an area where an amount of artificial or ambient light is controlled when images are taken by camera <NUM>, for example. Camera housing <NUM> may be formed by one or more walls at a location just above conveyor <NUM>. In some embodiments, camera <NUM> and camera housing <NUM> remain stationary relative to conveyor <NUM> and compartments <NUM> as animals move beneath camera <NUM> along path <NUM>.

Ultrasound transducers 106a, 106b, and 106c are configured to obtain ultrasound images of the animals in compartments <NUM> on conveyor <NUM>. In some embodiments, ultrasound transducers 106a, 106b, and 106c are configured to obtain an ultrasound image set of the animal that includes the ultrasound images. In some embodiments, an individual ultrasound transducer 106a, 106b, or 106c is configured to obtain one or more ultrasound images of a given animal on conveyor <NUM>. For example, the ultrasound images may include an ultrasound image set of an animal. The ultrasound image set may include one or more ultrasound images of the animal. If the ultrasound image set includes multiple ultrasound images, the multiple ultrasound images may be captured from different angles (e.g., a top view, side view, bottom view, etc.) and/or may be captured substantially simultaneously. The views may also include plan, elevation, and/or section views. The one or more views may create a standardized series of orthographic two-dimensional images that represent the form of the three-dimensional animal. For example, six views of the animal may be used, with each projection plane parallel to one of the coordinate axes of the animal. The views may be positioned relative to each other according to either a first-angle projection scheme or a third-angle projection scheme. The ultrasound images in the ultrasound image set may include separate images (e.g., images stored separately, but linked by a common identifier such as a serial number) or images stored together. An ultrasound image in an ultrasound image set may also be a composite image (e.g., an image created by cutting, cropping, rearranging, and/or overlapping two or more ultrasound images.

Ultrasound transducers 106a, 106b, and 106c are configured to obtain the ultrasound images while the ultrasound transducers 106a, 106b, and 106c move along a portion of path <NUM> with the animals. Ultrasound transducers 106a, 106b, and 106c are configured to move in at least two dimensions. The at least two dimensions comprise a first dimension along path <NUM> and a second dimension along a body of a given animal. The second dimension is substantially perpendicular to the first dimension and path <NUM>, for example. In some embodiments, an ultrasound transducer 106a, 106b, or 106c is configured to move in the first dimension and the second dimension substantially simultaneously while obtaining ultrasonic images, starting from the starting point.

Movement in two dimensions occurs at a controlled speed over defined distances that correspond to movement of an animal on conveyor <NUM>, and a length of a given animal. The controlled speed and distances facilitate acquisition of standardized images for each animal carried by conveyor <NUM>. A mechanical system comprising independent electrical linear actuators may be configured to move each ultrasound transducer 106a, 106b, and 106c along path <NUM>, or perpendicular to path <NUM>, along the body of a given fish <NUM>, or in other directions, for example. Such actuators may also be used to move a given transducer toward or away from the body of a fish <NUM> (e.g. to place pressure on the body of a fish <NUM> during a scan as described herein).

In some embodiments, a width of an ultrasonic transducer 106a, 106b, and 106c, and the movement in the second dimension defines an image area on the body of the animal. The image area includes target anatomy of the animal. In some embodiments, a width of an ultrasound transducer 106a, 106b, or 106c is at least <NUM>. In some embodiments, a width of an ultrasound transducer 106a, 106b, or 106c is at least <NUM>. In some embodiments, a width of an ultrasound transducer 106a, 106b, or 106c is at least <NUM>. In some embodiments, an ultrasound transducer 106a, 106b, or 106c is configured to move along the body of an animal over a distance of about <NUM>. In some embodiments, an ultrasound transducer 106a, 106b, or 106c is configured to move along the body of an animal over a distance of about <NUM>. In some embodiments, an ultrasound transducer 106a, 106b, or 106c is configured to move along the body of an animal over a distance of about <NUM>. It should be understood that the width of an ultrasound transducer and the distance that the ultrasound transducer moves along the length of the body of an animal may be adjusted based on the size or type of animal being scanned, for example.

In some embodiments, an ultrasound transducer 106a, 106b, or 106c is configured to contact an animal in a compartment <NUM> and keep pressure on the animal while ultrasound images are acquired. This may be accomplished through, for example, rolls located before and after a transducer. The rolls and transducer can be loaded with a spring or other systems to maintain steady pressure without damaging a fish.

By way of a non-limiting example, ultrasound transducers 106a, 106b, and 106c are configured to scan three fish <NUM> moving along conveyor <NUM> at the same time. The group of transducers is moving back and forth, to measure one group of fish <NUM> and then move for a next group of fish <NUM>. When the ultrasound scans are being performed, the transducers move at the same speed as the fish along path <NUM> of conveyor <NUM>. Movements of the group of transducers and conveyor <NUM> are synchronized. Independent electrical linear transducers are configured to move each ultrasound transducer 106a, 106b, and 106c along and perpendicular to path <NUM> along the body of a given fish <NUM> during ultrasound imaging. Such actuators may also be used to move a given transducer toward or away from the body of a fish <NUM> (e.g. to place pressure on the body of a fish <NUM> during a scan).

Sorter <NUM> is configured to sort the animals into groups. Sorter <NUM> is configured to sort the animals into groups as the animals come off of conveyor <NUM>. Sorter <NUM> comprises a mechanical arm controlled by control circuitry to move between multiple positions such that sorting the animal into a group comprises moving the mechanical arm to direct the animal from conveyor <NUM> to a same physical location as other animals in the group.

<FIG> illustrates a schematic view of system <NUM>, in accordance with one or more embodiments. <FIG> illustrates additional components of system <NUM> including control circuitry <NUM>. For example, <FIG> illustrates various wired or wireless electronic communication paths <NUM> formed between different components of system <NUM>. <FIG> illustrates lighting <NUM> that may be coupled to camera <NUM> or included in camera housing <NUM>, for example. <FIG> illustrates a gear motor <NUM> configured to drive conveyor <NUM>. <FIG> illustrates compression rollers <NUM> coupled to ultrasound transducers 106a, 106b, and 106c configured to place animals that are being ultrasonically imaged under pressure (e.g., to prevent unintentional movement during imaging), actuators <NUM> and <NUM> configured to move the ultrasound transducers (as described above), a compensation mechanism <NUM>, and a wagon <NUM> mechanically coupled to conveyor <NUM>. Compensation mechanism <NUM> is configured to adjust the pressure a transducer puts on a fish. Wagon <NUM> is configured to travel with the conveyor while a scan is performed with an ultrasound transducer.

Control circuitry <NUM> is configured to determine, based on the visual images from camera <NUM>, starting points on the animals for the ultrasound transducers. In some embodiments, control circuitry <NUM> is configured to control the ultrasound transducers to move along the portion of path <NUM> based on the starting point to obtain the ultrasound images. However, in some embodiments, the ultrasound transducers are controlled to move along the portion of path <NUM> by mechanical means. Control circuitry <NUM> is configured to determine the starting point on a given animal by providing the visual image(s) for that animal to a machine vision algorithm, which is trained to determine the starting point based on the visual image(s).

In some embodiments, determining a starting point comprises generating a pixel array based on the visual images or image set of the animal. The pixel array may refer to computer data that describes an image (e.g., pixel by pixel). In some embodiments, this may include one or more vectors, arrays, and/or matrices that represent either a Red, Green, Blue or grayscale image. Furthermore, in some embodiments, control circuity <NUM> may additionally convert the image set from a set of one or more vectors, arrays, and/or matrices to another set of one or more vectors, arrays, and/or matrices. For example, the control circuitry <NUM> may convert an image set having a red color array, a green color array, and a blue color to a grayscale color array. In some embodiments, for example, the animal is a fish and the starting point, determined based on the pixel array, corresponds to a start of an operculum of the fish.

Control circuitry <NUM> is configured to determine, based on the visual images and the ultrasound images, and optionally other information, characteristics of the animals. The control circuitry <NUM> is configured to receive the visual images from camera <NUM> (<FIG>) and the ultrasound images from ultrasound transducer 106a, 106b, or 106c (<FIG>). In some embodiments, control circuitry <NUM> also includes memory (as described herein), which may be incorporated into and/or accessible by control circuitry <NUM>. In some embodiments, control circuitry <NUM> may retrieve the (visual or ultrasound) image sets from memory.

In some embodiments, control circuitry <NUM> is configured to determine the starting point based on the RGB image set, and determine the characteristics based on the RGB image set and/or the ultrasound image set. A characteristic may be or describe a condition, feature, or quality of an animal, that may be used to sort an animal into a group. The characteristics may include a current physiological condition (e.g., a condition occurring normal in the body of the animal) such as a gender of the animal (e.g., as determined by the development of sex organs) and/or a stage of development in the animal (e.g., the state of smoltification in a fish). The characteristics may include a predisposition to a future physiological condition such as a growth rate, maturity date, and/or behavioral traits. The characteristics may include a pathological condition (e.g., a condition centered on an abnormality in the body of the animal based on a response to a disease) such as whether or not the animal is suffering from a given disease and/or is currently infected with a given disease. The characteristics may include a genetic condition (e.g., a condition based on the formation of the genome of the animal) such as whether or not the animal includes a given genotype. The characteristics may include a presence of a given biomarker (e.g., a measurable substance in an organism whose presence is indicative of a disease, infection, current internal condition, future internal condition, and/or environmental exposure). The characteristics may include phenotype characteristics (e.g., one or more observable characteristics of an animal resulting from the interaction of its genotype with the environment). These externally visible traits may include traits corresponding to physiological changes in the animal. For example, during smoltification in a fish (i.e., the series of physiological changes where juvenile salmonid fish adapt from living in fresh water to living in seawater), externally visible traits related to this physiological change may include altered body shape, increased skin reflectance (silvery coloration), and increased enzyme production (e.g., sodium-potassium adenosine triphosphatase) in the gills.

By way of several specific examples, the characteristics (which again may be determined based on ultrasound images, RGB images, and/or other information) may include a gender of an animal, presence of disease in an animal, size of an animal, early maturation of an animal, mature parr, presence of bacterial kidney disease in an animal, heart or skeletal muscle inflammation in an animal, a fat percentage of an animal, a size of an animal, a shape of an animal, a weight of an animal, and/or other characteristics. In some embodiments, the animal is a fish and the visual image is a red green blue (RGB) image. In some embodiments, control circuitry <NUM> is configured to determine, based on the RGB image, characteristics such as a short operculum in the fish and/or damage to gills of the fish, diseases resistance, growth performance, current diseases, smoltification status, and/or other characteristics (e.g., see other examples of characteristics described herein).

Control circuitry <NUM> is configured to determine the characteristics of an animal based on one or more ultrasound images and a visual image of that animal by inputting the one or more ultrasound images and a visual image to a machine learning mode such as an artificial neural network, which is trained to output the characteristics based on the one or more ultrasound (or visual) images. The artificial neural network is trained to identify one or more phenotype characteristics of the animal based on the ultrasound image, and determine presence of a biomarker in the animal indicative of the characteristic output by the artificial neural network based on the one or more phenotype characteristics.

As shown in <FIG>, control circuitry <NUM> may include various components configured to perform or control one or more of the operations described above, such as a programmable logic controller (PLC) input output (I/O) board <NUM> and an actuator <NUM> coupled to sorter <NUM>. Sorter <NUM> includes a mechanical arm <NUM> configured to move back and forth between different positions to sort the animals, for example. Control circuitry <NUM> may include a PLC <NUM>, an ultrasound transducer control board <NUM>, a human machine interface (HMI) <NUM>, and/or other components that are part of or configured to communicate with a computing system <NUM>, or other components.

By way of a non-limiting example, <FIG> illustrates a computing system <NUM> that is part of control circuitry <NUM> (<FIG>), featuring a machine learning model <NUM> configured to determine characteristics of animals, in accordance with one or more embodiments. As shown in <FIG>, system <NUM> may include client device <NUM>, client device <NUM> or other components. Each of client devices <NUM> and <NUM> may include any type of mobile terminal, fixed terminal, or other device. Each of these devices may receive content and data via input/output (hereinafter "I/O") paths and may also include processors and/or other components to send and receive commands, requests, and other suitable data using the I/O paths. Control circuitry <NUM> may comprise any suitable processing circuitry. Each of these devices may also include a user input interface and/or display for use in receiving and displaying data. By way of example, client devices <NUM> and <NUM> may include a desktop computer, a server, or other client device. Users may, for instance, utilize one or more client devices <NUM> and <NUM> to interact with one another, one or more servers, or other components of computing system <NUM>. It should be noted that, while one or more operations are described herein as being performed by particular components of computing system <NUM>, those operations may, in some embodiments, be performed by other components of computing system <NUM>. As an example, while one or more operations are described herein as being performed by components of client device <NUM>, those operations may, in some embodiments, be performed by components of client device <NUM>. It should be noted that, although some embodiments are described herein with respect to machine learning models, other prediction models (e.g., statistical models or other analytics models) may be used in lieu of or in addition to machine learning models in other embodiments (e.g., a statistical model replacing a machine learning model and a non-statistical model replacing a non-machine-learning model in one or more embodiments).

Each of these devices may also include memory in the form of electronic storage. The electronic storage may include non-transitory storage media that electronically stores information. The electronic storage media of the electronic storages may include one or both of (i) system storage that is provided integrally (e.g., substantially non-removable) with servers or client devices or (ii) removable storage that is removably connectable to the servers or client devices via, for example, a port (e.g., a USB port, a firewire port, etc.) or a drive (e.g., a disk drive, etc.). The electronic storages may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media. The electronic storages may include one or more virtual storage resources (e.g., cloud storage, a virtual private network, and/or other virtual storage resources). The electronic storage may store software algorithms, information determined by the processors, information obtained from servers, information obtained from client devices, or other information that enables the functionality as described herein.

<FIG> also includes communication paths <NUM>, <NUM>, and <NUM>. Communication paths <NUM>, <NUM>, and <NUM> may include a local network (e.g., a Wi-Fi or other wired or wireless local network), the Internet, a mobile phone network, a mobile voice or data network (e.g., a <NUM> or LTE network), a cable network, a public switched telephone network, wires, or other types of communications network or combinations of communications networks. Communication paths <NUM>, <NUM>, and <NUM> may separately or together include one or more communications paths, such as a satellite path, a fiber-optic path, a cable path, a path that supports Internet communications (e.g., IPTV), free-space connections (e.g., for broadcast or other wireless signals), or any other suitable wired or wireless communications path or combination of such paths. The computing devices may include additional communication paths linking a plurality of hardware, software, and/or firmware components operating together. For example, the computing devices may be implemented by a cloud of computing platforms operating together as the computing devices.

In some embodiments, computing system <NUM> may use one or more prediction models to predict characteristics based on visual images, ultrasound images, or other information. For example, as shown in <FIG>, computing system <NUM> may predict a characteristic of an animal (e.g., a fish identified by a specimen identification) using machine learning model <NUM>. The determination may be output shown as output <NUM> on client device <NUM>. Computing system <NUM> may include one or more neural networks (e.g., as discussed in relation to <FIG>) or other machine learning models. These neural networks or other machine learning models may be located locally (e.g., executed by one or more components of computing system <NUM> located at or near fish processing) or remotely (e.g., executed by a remote or cloud server that is part of computing system <NUM>).

As an example, with respect to <FIG>, machine learning model <NUM> may take inputs <NUM> and provide outputs <NUM>. The inputs may include multiple data sets such as a training data set and a test data set. The data sets may represent (e.g., ultrasound) images (or image sets) of animals such as fish or other animals. In one use case, outputs <NUM> may be fed back to machine learning model <NUM> as input to train machine learning model <NUM> (e.g., alone or in conjunction with user indications of the accuracy of outputs <NUM>, labels associated with the inputs, or with other reference feedback information). In another use case, machine learning model <NUM> may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of its prediction (e.g., outputs <NUM>) and reference feedback information (e.g., user indication of accuracy, reference labels, or other information). In another use case, where machine learning model <NUM> is a neural network, connection weights may be adjusted to reconcile differences between the neural network's prediction and the reference feedback. In a further use case, one or more neurons (or nodes) of the neural network may require that their respective errors are sent backward through the neural network to them to facilitate the update process (e.g., backpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the machine learning model <NUM> may be trained to generate better predictions.

Machine learning model <NUM> may be trained to detect the characteristics in animals based on a set of ultrasound images. For example, ultrasound transducers 106a, 106b, or 106c (<FIG>) may generate the ultrasound image set of a first fish (as an example of an animal). Computing system <NUM> may determine a genotype biomarker in the first fish. The presence of a particular genotype biomarker is then correlated to one or more phenotype characteristics. For example, machine learning model <NUM> may have classifications for characteristics. Machine learning model <NUM> is then trained based on a first data set (e.g., including data of the first fish and others) to classify a specimen as having a given characteristic when particular ultrasound image features are present.

The system may then receive an ultrasound image set of a second fish. Computing system <NUM> may input one or more of the ultrasound images in the set into machine learning model <NUM>. Computing system <NUM> may then receive an output from machine learning model <NUM> indicating that the second fish has the same characteristic (e.g., genotype biomarker) as the first. For example, computing system <NUM> may input a second data set (e.g., ultrasound image sets of fish for which characteristics are not known) into machine learning model <NUM>. Machine learning model <NUM> may then classify the image sets of fish based on the images.

<FIG> shows a graphical representations of artificial neural network models for characteristic determination based on (visual or) ultrasound images, in accordance with one or more embodiments. Model <NUM> illustrates an artificial neural network. Model <NUM> includes input layer <NUM>. Ultrasound image sets may be entered into model <NUM> at this level. Model <NUM> also includes one or more hidden layers (e.g., hidden layer <NUM> and hidden layer <NUM>). Model <NUM> may be based on a large collection of neural units (or artificial neurons). Model <NUM> loosely mimics the manner in which a biological brain works (e.g., via large clusters of biological neurons connected by axons). Each neural unit of a model <NUM> may be connected with many other neural units of model <NUM>. Such connections can be enforcing or inhibitory in their effect on the activation state of connected neural units. In some embodiments, each individual neural unit may have a summation function which combines the values of all of its inputs together. In some embodiments, each connection (or the neural unit itself) may have a threshold function such that the signal must surpass before it propagates to other neural units. Model <NUM> may be self-learning and trained, rather than explicitly programmed, and can perform significantly better in certain areas of problem solving, as compared to traditional computer programs. During training, output layer <NUM> may corresponds to a classification of model <NUM> (e.g., whether or not a given image set corresponds to a characteristic) and an input known to correspond to that classification may be input into input layer <NUM>. In some embodiments, model <NUM> may include multiple layers (e.g., where a signal path traverses from front layers to back layers). In some embodiments, back propagation techniques may be utilized by model <NUM> where forward stimulation is used to reset weights on the "front" neural units. In some embodiments, stimulation and inhibition for the model may be more free-flowing, with connections interacting in a more chaotic and complex fashion. Model <NUM> also includes output layer <NUM>. During testing, output layer <NUM> may indicate whether or not a given input corresponds to a classification of model <NUM> (e.g., whether or not a given image set corresponds to a characteristic).

<FIG> also illustrates model <NUM>, which is a convolutional neural network. The convolutional neural network is an artificial neural network that features one or more convolutional layers. Convolution layers extract features from an input image. Convolution preserves the relationship between pixels by learning image features using small squares of input data. As shown in model <NUM>, input layer <NUM> may proceed to convolution blocks <NUM> and <NUM> before being output to convolutional output or block <NUM>. In some embodiments, model <NUM> may itself serve as an input to model <NUM>.

In some embodiments, model <NUM> may implement an inverted residual structure where the input and output of a residual block (e.g., block <NUM>) are thin bottleneck layers. A residual layer may feed into the next layer and directly into layers that are one or more layers downstream. A bottleneck layer (e.g., block <NUM>) is a layer that contains few neural units compared to the previous layers. Model <NUM> may use a bottleneck layer to obtain a representation of the input with reduced dimensionality. An example of this is the use of autoencoders with bottleneck layers for nonlinear dimensionality reduction. Additionally, model <NUM> may remove non-linearities in a narrow layer (e.g., block <NUM>) in order to maintain representational power. In some embodiments, the design of model <NUM> may also be guided by the metric of computation complexity (e.g., the number of floating point operations). In some embodiments, model <NUM> may increase the feature map dimension at all units to involve as many locations as possible instead of sharply increasing the feature map dimensions at neural units that perform downsampling. In some embodiments, model <NUM> may decrease the depth and increase width of residual layers in the downstream direction.

Returning to <FIG> and <FIG>, control circuitry <NUM> (e.g., including actuator <NUM> and PLC board <NUM> shown in <FIG>) is configured to control sorter <NUM> to sort the animals into the groups based on the characteristics. Control circuitry <NUM> is configured to cause sorter <NUM> to handle, sort, and/or transfer animals (e.g., for vaccination, gender segregation, transfer to sea or breeding area, etc.). The characteristics are detected based on the visual and ultrasound images in real-time (e.g., as the animals are transported along conveyor <NUM> or otherwise transferred). That is, following the output of a given characteristic (e.g., a genotype biomarker) for a given animal, sorter <NUM> sorts (e.g., actuator <NUM> may cause a mechanical arm <NUM> to move back and forth between different positions to sort) the animal based on the determined characteristic.

<FIG> illustrates a method <NUM> for sorting animals with a sorting system. Method <NUM> may be executed by a system such as system <NUM> (<FIG>, <FIG>) and/or other systems. System <NUM> comprises a conveyor, a camera, an ultrasound transducer, a sorter comprising a mechanical arm, and control circuitry. The operations of method <NUM> presented below are intended to be illustrative. In some embodiments, method <NUM> may be accomplished with one or more additional operations not described, and/or without one or more of the operations discussed. Additionally, the order in which the operations of method <NUM> are illustrated in <FIG> and described below is not intended to be limiting.

In some embodiments, method <NUM> may be implemented, at least in part, in one or more processing devices such as one or more processing devices described herein (e.g., a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information). The one or more processing devices may include one or more devices executing some or all of the operations of method <NUM> in response to instructions (e.g., machine readable instructions) stored electronically on an electronic storage medium. The one or more processing devices may include one or more devices configured through hardware, firmware, and/or software to be specifically designed for execution of one or more of the operations of method <NUM>.

At an operation <NUM>, animals are received with a plurality of compartments of the conveyor, and moved along a path. In some embodiments, operation <NUM> is performed by a conveyor the same as or similar to conveyor <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, a visual image (or set of visual images) of an animal in a compartment on the conveyor is obtained as the animal moves past a camera. In some embodiments, the camera is configured to obtain a red green blue (RGB) image set that includes the visual image. In some embodiments, operation <NUM> is performed by camera the same as or similar to camera <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, an ultrasound image (or set of ultrasound images) of the animal is obtained with an ultrasound transducer. The ultrasound image of the animal is obtained with the animal in the compartment on the conveyor. The ultrasound transducer is configured to obtain the ultrasound image while the ultrasound transducer moves along a portion of the path with the animal. In some embodiments, the ultrasound transducer is configured to obtain an ultrasound image set of the animal that includes the ultrasound image. In some embodiments, operation <NUM> is performed by one or more ultrasound transducers the same as or similar to ultrasound transducers 106a, 106b, or 106c (shown in <FIG> and described herein).

At an operation <NUM>, a starting point on the animal is determined for the ultrasound transducer, with the control circuitry, based on the visual image. The control circuitry is configured to determine the starting point on the animal by providing the visual image to a machine vision algorithm, which is trained to determine the starting point based on the visual image. In some embodiments, the animal is a fish and the starting point corresponds to a start of an operculum of the fish.

Operation <NUM> also includes controlling the ultrasound transducer to move along the portion of the path based on the starting point to obtain the ultrasound image. Controlling the ultrasound transducer comprises controlling the ultrasound transducer to move in at least two dimensions. The at least two dimensions comprise a first dimension along the path and a second dimension along a body of the animal. The second dimension is substantially perpendicular to the first dimension and the path. The ultrasound transducer is configured to move in the first dimension and the second dimension substantially simultaneously while obtaining the ultrasonic image, starting from the starting point. A width of the ultrasonic transducer and the movement in the second dimension defines an image area on the body of the animal. The image area includes a target anatomy of the animal. In some embodiments, the sorting system includes a plurality of ultrasound transducers, and operation <NUM> includes controlling, with the control circuitry, the plurality of ultrasound transducers to obtain ultrasound images of a plurality of animals in a plurality of compartments on the conveyor at the same time. In some embodiments, operation <NUM> is performed by control circuitry the same as or similar to control circuitry <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, a characteristic of the animal is determined. The characteristic is determined with the control circuitry, based on the ultrasound image. The characteristic is gender of the animal, presence of disease in the animal, size of the animal, early maturation of the animal, mature parr, presence of bacterial kidney disease in the animal, heart and/or skeletal muscle inflammation in the animal, a fat percentage of the animal, and/or other characteristics.

The control circuitry is configured to determine the characteristic of the animal based on the ultrasound image by inputting the ultrasound image to an artificial neural network, which is trained to output the characteristic based on the ultrasound image. The artificial neural network is trained to identify one or more phenotype characteristics of the animal based on the ultrasound image, and determine presence of a biomarker in the animal indicative of the characteristic output by the artificial neural network based on the one or more phenotype characteristics. In some embodiments, the control circuitry is configured to determine the starting point based on the RGB image set, and determine the characteristic based on the ultrasound image set. In some embodiments, the animal is a fish and the visual image is a red green blue (RGB) image, and operation <NUM> comprises determining, with the control circuitry, based on the RGB image, a short operculum in the fish and/or damage to gills of the fish, diseases resistance, growth performance, current diseases, and/or other characteristics. In some embodiments, operation <NUM> is performed by control circuitry the same as or similar to control circuitry <NUM> (shown in <FIG> and described herein).

At an operation <NUM>, the sorter is controlled to sort the animal into a group based on the characteristic. The sorter comprises a mechanical arm. The mechanical arm is controlled, with the control circuitry, to move between multiple positions such that sorting the animal into a group comprises moving the mechanical arm to direct the animal from the conveyor to a same physical location as other animals in a group. In some embodiments, operation <NUM> is performed by control circuitry the same as or similar to control circuitry <NUM> (shown in <FIG> and described herein) or a sorter the same as or similar to sorter <NUM> (shown in <FIG> and described herein).

In block diagrams such as <FIG>, illustrated components are depicted as discrete functional blocks, but embodiments are not limited to systems in which the functionality described herein is organized as illustrated. The functionality provided by each of the components may be provided by software or hardware modules that are differently organized than is presently depicted, for example such software or hardware may be intermingled, conjoined, replicated, broken up, distributed (e.g. within a data center or geographically), or otherwise differently organized. The functionality described herein may be provided by one or more processors of one or more computers executing code stored on a tangible, non-transitory, machine readable medium. In some cases, notwithstanding use of the singular term "medium," the instructions may be distributed on different storage devices associated with different computing devices, for instance, with each computing device having a different subset of the instructions, an implementation consistent with usage of the singular term "medium" herein. In some cases, third party content delivery networks may host some or all of the information conveyed over networks, in which case, to the extent information (e.g., content) is said to be supplied or otherwise provided, the information may be provided by sending instructions to retrieve that information from a content delivery network.

It is contemplated that the steps or descriptions of <FIG> may be used with any other embodiment of this disclosure. In addition, the steps and descriptions described in relation to <FIG> may be performed in alternative orders or in parallel to further the purposes of this disclosure. For example, each of these steps may be performed in any order or in parallel or substantially simultaneously to reduce lag or increase the speed of the system or method. Furthermore, it should be noted that any of the devices or equipment discussed in relation to <FIG> could be used to perform one or more of the steps in <FIG>.

To the extent that it aids understanding of the concepts described above, <FIG> illustrates a view of system <NUM> from a different angle than the view shown in <FIG>. <FIG> illustrates different views <NUM>, <NUM>, <NUM>, and <NUM> of system <NUM> with a hypothetical operator <NUM> shown at system <NUM> in each view. View <NUM> is a front view. View <NUM> is an end view. View <NUM> is a perspective view from an angle above system <NUM>. View <NUM> is a top view. In <FIG> and <FIG>, like reference numerals illustrate like components (which are described above).

Other embodiments of the present system are contemplated. <FIG> illustrate examples of other possible embodiments of the present system. System components shown in <FIG> are similar to and/or the same as those described above with respect to <FIG>. As such, the descriptions of components described above also apply here.

For example, <FIG> illustrates a single ultrasonic transducer <NUM> version of the sorting system where animals travel down a conveyor <NUM> and are sorted via doors <NUM>, <NUM>, or by simply falling off an end <NUM> of conveyor <NUM>. <FIG> also illustrates a feeder <NUM>, a camera <NUM>, and an actuator <NUM>.

<FIG> illustrates a schematic view of the version of the system shown in <FIG>. <FIG> illustrates additional components of the system including control circuitry <NUM>. For example, <FIG> illustrates various wired or wireless electronic communication paths <NUM> formed between different components of the system. <FIG> illustrates lighting <NUM> that may be coupled to camera <NUM>, for example. <FIG> illustrates a gear motor <NUM> configured to drive conveyor <NUM>. <FIG> illustrates compression rollers <NUM> coupled to ultrasound transducer <NUM> configured to place animals that are being ultrasonically imaged under pressure (e.g., to prevent unintentional movement during imaging), actuators <NUM> and <NUM> configured to move the ultrasound transducer (as described above), a compensation mechanism <NUM>, sorting door actuators <NUM> and <NUM>, computing system <NUM>, a programmable logic controller (PLC) input output (I/O) board <NUM>, a motion controller <NUM>, an ultrasound transducer control board <NUM>, a human machine interface (HMI) <NUM>, and/or other components that are part of or configured to communicate with a computing system <NUM>, or other components.

<FIG> illustrates several views <NUM> - <NUM> of a three (in this example) conveyor embodiment of the present system. Each conveyor in the embodiment shown in <FIG> may be similar to the conveyor shown in <FIG> and <FIG>, for example (e.g., with a conveyor, doors, a feeder, a camera, control circuitry, etc.).

Claim 1:
A system for sorting animals, comprising:
a conveyor (<NUM>) configured to receive animals and move the animals along a path (<NUM>);
an ultrasound transducer (106a, 106b, 106c) configured to obtain an ultrasound image of an animal (<NUM>) located on the path (<NUM>), the ultrasound transducer (106a, 106b, 106c) configured to obtain the ultrasound image while the animal moves along a portion of the path (<NUM>);
a camera (<NUM>) configured to obtain a visual image of the animal;
a sorter (<NUM>) configured to sort the animal into a group; and
control circuitry (<NUM>) configured to:
determine, based on the visual image and the ultrasound image, characteristics of the animal; and
control the sorter (<NUM>) to sort the animal into the group based on the characteristics;
wherein the sorter (<NUM>) comprises a mechanical arm (<NUM>) controlled by the control circuitry (<NUM>) to move between multiple positions such that sorting the animal into a group comprises moving the mechanical arm to direct the animal from the conveyor (<NUM>) to a same physical location as other animals in the group.