Computer Vision and Feeding Behavior Based
Intelligent Feeding Controller for Fish in Aquaculture

Chao Zhou, Kai Lin, Daming Xu, Chuanheng Sun, Lan Chen, Song Zhang, Qiang Guo

► To cite this version:

Chao Zhou, Kai Lin, Daming Xu, Chuanheng Sun, Lan Chen, et al. Computer Vision and Feeding Behavior Based Intelligent Feeding Controller for Fish in Aquaculture. 11th International Conference on Computer and Computing Technologies in Agriculture (CCTA), Aug 2017, Jilin, China. pp.98-107, 10.1007/978-3-030-06137-1_10. hal-02124258

HAL Id: hal-02124258

https://inria.hal.science/hal-02124258

Submitted on 9 May 2019

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Computer Vision and Feeding Behavior Based
Intelligent Feeding Controller for Fish in Aquaculture

Chao Zhou<sup>1,2,3,4(✉)</sup>, Kai Lin<sup>1,2,3</sup>, Daming Xu<sup>1,2,3</sup>, Chuanheng Sun<sup>1,2,3</sup>, Lan Chen<sup>1,2,3</sup>, Song Zhang<sup>1,2,3</sup>, Qiang Guo<sup>1,2,3</sup>

<sup>1</sup>Beijing Research Center for Information Technology in Agriculture, Beijing 100097, China
{zhouc,link, xudm, sunch}@nercita.org.cn,
{chenlan199366, 1148640196, 865011433}@qq.com

<sup>2</sup>National Engineering Research Center for Information Technology in Agriculture, Beijing 100097, China

<sup>3</sup> National Engineering Laboratory for Agri-product Quality Traceability, Beijing, 100097, China

<sup>4</sup>School of Automation, Beijing Institute of Technology, Beijing 100081, China

**Abstract.** In aquaculture, the feeding technology determined the feed conversion rate and cost. However, the intelligence of existing feeding devices is not very high. they can't change the amount of feed according to the fish appetite automatically. In order to solve the above issues, in this paper, a feeding controller based on machine vision and feeding behavior was designed on the basis of the original feeder. The hardware platform was built on the I.MX6 microcontroller, and the software was designed via the embedded Linux OS. Moreover, the feeding behavior analysis and automatic feeding control method based on image processing were also studied. Firstly, the images of fish feeding process were collected and analyzed. Then the Delaunay Triangulation was used to extract the feeding behavior parameter FIFFB (flocking index of fish feeding behavior). Finally, the feeding decision was made according to the defined threshold. Compared with the traditional feeder, the controller designed in this paper is more intelligent and can reduce feed waste. Meanwhile, water pollution also can be reduced. The automatic feeding control was realized during feeding process.

**Keywords:** Computer Vision. Feeding behavior. Intelligent Control. Aquaculture

# 1 Introduction

Feed is one of the most important variable cost in aquaculture, and it can even reach 80% in some cultured species[1-3]. So it is important to minimum the feed cost [4]. It not only reduces the feed waste and save costs, but also reduces the environmental pollution caused by residual bait[5, 6]. When the feeding amount is too less, the growth rate will became slow and the breeding cycle will be prolong, resulting in the decrease of unit catches[7].On the other hand, there are many factors exists that can be affect the feeding rate of fish, including environmental factors, physiological factors, nutritional factors and management factors, etc [4]. These factors lead to uncertainty in the amount of feed required offish, and it is almost impossible to obtain accurate quantity by calculation[8]. The operator is often difficult to accurately grasp the most suitable feeding level to meet the needs of fish. Therefore, it is urgent to achieve automatic control of feeding process in aquaculture[9, 10].

The traditional automatic feeder can only fed a fixed amount at the setting time according to the preset program. And it is only a substitute for throwing bait by manual. It does not make corresponding changes in feeding amount and time
according to the fish appetite. In fact, it does not achieve the automatic feeding [11].

The deep understanding of fish behavior shows that although there are many factors affecting the feed intake, all the factors can be reflected by their behavioral changes. In the feeding process, the speed and direction of fish will be changed at different stages [12-14]. In recent years, according to the feeding behavior change, the feeding amount can be adjusted has become a new research focus. As the continuous development of machine vision technology, because of its advantages of non-contact, low cost and simple development, it has become the first choice for fish feeding behavior detection and analysis[15-19]. As in most fish farms exists many issues such as high humidity, wide range of temperature changes, the traditional PC are not suitable for such occasions. While the embedded control technology is very suitable for such abominable environment. The feeding process can be adjusted and fed back by various feeding behavior detection sensors in real time, and make they has the ability of automatically determine the feed demand of the fish. It is considered as an intelligent feeding system with high adaptability.

In this paper, a feeding controller was designed on the basis of the existing feeding machine, and the original structure of feeding machine was not modified. The controlling of feeding amount, time and frequency of the existing feeding machine were achieved through a variety of interfaces and protocols. The feeding behavior of fish was analyzed by machine vision and embedded control technology, and the feed rate and the starting or stopping time of feeding machine were adjusted in real time. The intelligent control of feeding process was realized to reduce feed consumption and water pollution.

## 2 Materials and methods

Embedded vision has mostly been based on DSP or FPGA platforms over the past 20 years. However, there are many problems. The DSP usually has low computing speed. Therefore, it is necessary to design special DSP to achieve complex operations with multi-cores, parallel processing mechanisms, or optimize algorithms for special platforms. Therefore, the cost of developing visual application modules based on the DSP platform is higher. In addition, DSP and FPGA are suitable for parallel operations. Statements containing jump instructions are inefficient, such as "if"... else...".

With the development of ARM technology, ARM platform has brought unprecedented opportunities for the development and application of embedded visual application module. It does not need quadratic optimizations, and only needs to write efficient image processing algorithms on the PC platform. After cross compiling, we can get a visual image processing algorithm which can run efficiently on the ARM platform. Through the support of QT, the processed image and target information are displayed in the form of visual interface, so it has good interaction.

The feeding controller system designed in this paper was based on embedded Linux OS and the ARM Cortex A9 kernel micro-controller. Among them, the main control module of the feeding controller is the key part of the design. The controller collects the feeding image of fish in the breeding area via the industrial camera. And the collected images were analyzed and processed. According to the processing result, the feeding controller sends control instructions to the feeding machine to realize automatic feeding according to the requirement. The automatic feeding controller designed in this paper was shown in Fig. 1.
<figure><img src="image_2.png" /><figcaption>Fig. 1. The schematic diagram of system</figcaption></figure>

## 2.1 Hardware Platform Design

The main control chip in this paper was the i.MX6 Quad, it is an important product of NXP Semiconductor in image and video processor. It has the lowest power consumption on the basis of providing high performance processing capability. It is implemented by four 1GHz ARM™ Cortex-A9 core processors. i.MX 6 series can provide eye-catching multimedia performance. And its high image processing capabilities can support high-definition video operation. They include 2D and 3D graphics processors; Three hardware accelerated graphics processing units (GPU); 3D 1080p video processing, support full duplex video image processing in multiple formats without CPU resources. Each processor provides interfaces for connecting to peripherals devices, such as WLAN, Ethernet, etc [20].

The embedded hardware platform of this topic also includes the interface circuit of feeding machine (Ethernet, USB, IO, UART), human computer interface (mouse, monitor, camera), storage interfaces (SD, SDRAM, NORFLASH, NANDFLASH) and minimal system circuits (JTAG, RTC, etc.). The hardware design of the system was shown in Fig. 2.

<figure><img src="image_3.png" /><figcaption>Fig. 2. The schematic diagram of system hardware.</figcaption></figure>
**Design of image acquisition circuit.** QSC200 industrial camera was used to image acquisition, 1/3 "CMOS image sensor, video resolution 640*480 (15fps), 6-12mm manual zoom industrial lens, angle of view: 53°-28°. The industrial camera was connected to the main controller through the USB interface. It can collect the image and transfer it back to the controller for processing.

**Design of communication interface circuit for feeding machine.** The interface of the feeding controller is the key part of this study, which is related to whether the feeding controller can be used normally. Through the investigation of most feeding machines in the market, the reserved or available interfaces are basically IO port, UART, Ethernet interface and so on. In order to be compatible with the feeder, it is necessary to take into account all the interfaces above, and to set aside interfaces such as WLAN, 4G and so on. The two UART interfaces use the MAX3232 to realize the level conversion of the controller and feeder. The Ethernet interface was implemented by RTL8211E-VB-CG chip, which provides faster and smoother speed for onboard Gigabit Ethernet. The USB controller was used to implement the USB interface. The WLAN interface was implemented through the S500M module.

## 2.2 Software Platform Design

In addition, the current embedded operating system such as Linux, which can support complex data processing, is the key to real-time decision-making [21, 22] or tracking system [23]. Therefore, it is very suitable for the development of mobile devices [24].

The design of the feeding system software is mainly based on the Linux OS platform. Before writing the acquisition, display, transmission and identification program of feeding image, it is necessary to build a programming environment. In this paper, Qt image interface system was used to develop visualization and operation interface of feeding controller. Image processing vision library Open CV was also cross platform. In order to solve the problem of compatibility problem when the software is transplanted, we must build a cross platform development environment as follows.

Development environment build. Developing ARM-Linux image processing programs requires three systems as shown in Fig. 3. Windows OS is mainly used for the preparation of program code. The PC-Linux virtual machine OS is primarily used to cross compile the written program code; the ARM-Linux OS is used to run compiled executable files.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Development environment build</figcaption></figure>
After building Windows OS, PC-Linux virtual machine OS and ARM-Linux OS, the developing of embedded applications can be started. The complete embedded program cross compiling and developing process was shown in Fig. 4.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Embedded program cross compiling development process</figcaption></figure>

**Embedded image processing visual library (Open CV).** OpenCV, like Qt, is also cross platform and supports both Windows and Linux, which makes the development process a lot of convenience[21]. The flowchart of the library based on Open CV was shown in Fig. 5.

<figure><img src="image_6.png" /><figcaption>Fig. 5. The flowchart of the processing library</figcaption></figure>
The extraction method of feeding behavior index used in this paper are referred to literatures [22] and [23]. The fish target was extracted by using some image processing algorithms, such as background extraction [24]. During the feeding process, there is often overlap due to the fish activities. In order to solve this problem, this paper uses the watershed algorithm for the segmentation of overlapping targets [25, 26], and gets the image of the individual fish. Then the non-fish targets were filtered out, and the big or small targets were filtered out to get the binary image of the fish. In this paper, the position of the fish was expressed by the centroid. And the calculation of centroid was realized by the order moments [27].

Finally, the method in literature [23] was used to extract behavioral indicators during fish feeding. By using the Delaunay Triangulation, the behavior index FIFFB can be obtained [28, 29].

# 3 Results and discussion

The experiment was performed at Xiaotangshan National Experiment Station for Precision Agriculture, Beijing, China. The *Cyprinus carpio* var. *specularis* was selected as the subjects, which had been kept in the tank for a month and thought they had been adapted to the current environment. After the experiment, they were fed with commercial feed. In each feeding, the feed was supplied until there are more pellets floating on the water, and most fish do not eat the feed and spread out, then stop feeding and remove the remaining pellets. During feeding, the feeding image of fish were collected, analyzed and processed.

Fig. 6a is a binary image of fish feeding activity after image processing, and the centroid represents the spatial distribution of fish. And Fig. 6b is the process of Delaunay Triangulation. After triangulating the centroid of the fish in each frame of the feeding process, the FIFFB of the whole feeding process can be obtained, as shown in Fig. 7. Results shown that after feeding (t>10), the change of FIFFB is obvious, and shows a certain change rule. When the fish snatched violently, the FIFFB becomes smaller, and vice versa. After many trials and manual observation, the threshold for stop feeding was chosen as: FIFFB=595. Meanwhile, the controller sends stop feeding instructions to the feed machine.

<figure><img src="image_7.png" /><figcaption>Fig. 6. Delaunay Triangulation</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Fig. 7. FIFFB change and threshold selection.</figcaption></figure>

In order to verify the feeding decision accuracy and performance of this threshold, the decision output results were compared with the results observed by humans. The basis for manual observation is that: when the fish has almost no reaction to the feed and swims around, stop feeding, otherwise, continue feeding.

In the experiment, ten feedings were performed. When the stop feeding decision was made by expert observation, the value of FIFFB was calculated at this time, as shown in Table 1. In order to evaluate the consistency between threshold and artificial judgement, the deviation degree was used to evaluate its performance, which can be defined as:

$$D = \frac{|A - X|}{A} \qquad (1)$$

Where A is the target data, X is actual. It can be seen that the average value is 593.5, and the maximum deviation is 2.5%, which is very close to the threshold, indicating that the consistency of the two data sets is better. Therefore, it can be used as a substitute for artificial feeding.

<table><caption>**Table 1.** The FIFFB value of the 10 times feeding</caption><thead><tr><th>Number</th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>7</th><th>8</th><th>9</th><th>10</th></tr></thead><tbody><tr><th>FIFFB</th><td>595</td><td>590</td><td>601</td><td>582</td><td>610</td><td>594</td><td>599</td><td>603</td><td>585</td><td>587</td></tr></tbody></table>

## 4 Conclusion

In this paper, a feeding controller for cultured fish based on computer vision and feeding behavior was designed and implemented. At the same time, the hardware and software platform of feeding controller based on I.MX6Q microcontroller and embedded Linux OS were completed. The feeding images can be acquired and analyzed in real time. The feeding behavior of fish was quantified by Delaunay Triangulation, and the threshold of stop feeding was obtained. Compared with the traditional feeding machine, the feeding controller designed in this paper can realize the automatic intelligent control in the feeding process of fish, which plays an important role in improving the economic efficiency and automation level in the process of raising fish. However, only the *Cyprinus carpio* var. *specularis* was used in the experiment. The next step will be to validate the controller with other species to apply more varieties.

**Acknowledgments:** The research in this paper was supported by the National Key Research and Development Program of China (2017YFD0701705) and the Beijing Excellent Talents Development Project (2017000057592G125).

## References

1. Rola, W.R. and Hasan, M.R., Economics of aquaculture feeding practices: a synthesis of case studies undertaken in six Asian countries, FAO, ROME. (2007)

2. Huynh, T.-G., Shiu, Y.-L., Nguyen, T.-P. et al.: Current applications, selection, and possible
mechanisms of actions of synbiotics in improving the growth and health status in aquaculture: A review. Fish Shellfish Immunol 64, 367-382 (2017)

3. Hasan, M.R. and New, M.B.: On-farm feeding and feed management in aquaculture. Food & Agriculture Organization of the United Nations, (2014)

4. Sun, M., Hassan, S.G. and Li, D.: Models for estimating feed intake in aquaculture: A review. Comput Electron Agric 127, 425-438 (2016)

5. Føre, M., Alfredsen, J.A. and Gronningsater, A.: Development of two telemetry-based systems for monitoring the feeding behaviour of Atlantic salmon (Salmo salar L.) in aquaculture sea-cages. Comput Electron Agric 76 (2), 240-251 (2011)

6. Chang, C.M., Fang, W., Jiao, R.C. et al.: Development of an intelligent feeding controller for indoor intensive culturing of eel. Aquacult Eng 32 (2), 343-353 (2005)

7. Wu, T.-H., Huang, Y.-I. and Chen, J.-M.: Development of an adaptive neural-based fuzzy inference system for feeding decision-making assessment in silver perch (Bidyanus bidyanus) culture. Aquacult Eng 66, 41-51 (2015)

8. Cho, C.Y., Hynes, J.D., Wood, K.R. et al.: Development of high-nutrient-dense, low-pollution diets and prediction of aquaculture wastes using biological approaches. Aquaculture 124 (1), 293-305 (1994)

9. Soto-Zarazúa, G.M., Rico-García, E., Ocampo, R. et al.: Fuzzy-logic-based feeder system for intensive tilapia production ( Oreochromis niloticus ). Aquacult Int 18 (3), 379-391 (2010)

10. Zhou, C., Lin, K., Xu, D. et al.: Near infrared computer vision and neuro-fuzzy model-based feeding decision system for fish in aquaculture. Comput Electron Agric 146, 114-124 (2018)

11. Zhou, C., Xu, D., Lin, K. et al., Intelligent feeding control methods in aquaculture with an emphasis on fish: a review, Reviews in Aquaculture. (2017)

12. Pinkiewicz, T.H., Purser, G.J. and Williams, R.N.: A computer vision system to analyse the swimming behaviour of farmed fish in commercial aquaculture facilities: A case study using cage-held Atlantic salmon. Aquacult Eng 45 (1), 20-27 (2011)

13. Martins, C.I., Galhardo, L., Noble, C. et al.: Behavioural indicators of welfare in farmed fish. Fish Physiol Biochem 38 (1), 17 (2012)

14. Wyatt, T.: Some effects of food density on the growth and behaviour of plaice larvae. Mar Biol 14 (3), 210-216 (1972)

15. Xu, J., Liu, Y., Cui, S. et al.: Behavioral responses of tilapia (Oreochromis niloticus) to acute fluctuations in dissolved oxygen levels as monitored by computer vision. Aquacult Eng 35 (3), 207-217 (2006)

16. Zhao, J., Gu, Z., Shi, M. et al.: Spatial behavioral characteristics and statistics-based kinetic energy modeling in special behaviors detection of a shoal of fish in a recirculating aquaculture system. Comput Electron Agric 127, 271-280 (2016)

17. Ye, Z., Zhao, J., Han, Z. et al.: Behavioral Characteristics and Statistics-Based Imaging Techniques in the Assessment and Optimization of Tilapia Feeding in a Recirculating Aquaculture System. Transactions of the ASABE 59 (1), 345-355 (2016)

18. Zhou, C., Yang, X., Zhang, B. et al.: An adaptive image enhancement method for a recirculating aquaculture system. Sci Rep 7 (1), 6243 (2017)

19. Zhou, C., Sun, C., Lin, K. et al., Handling Water Reflections for Computer Vision in Aquaculture, Transactions of the ASABE. (2018)

20. Semiconductors, N. i.MX 6Dual/6Quad Applications Processors for Industrial Products. http://cache.nxp.com/assets/documents/data/en/data-sheets/IMX6DQIEC.pdf?fsrch=1&sr=2&pageNum=1, (2017)

21. Coombs, J., Prabhu, R. and Peake, G.: Overcoming the challenges of porting OpenCV to TI's embedded ARM + DSP platforms. International Journal of Electrical Engineering Education 49 (3), 260-274 (2012)

22. Pautsina, A., Císaŕ, P., Štys, D. et al.: Infrared reflection system for indoor 3D tracking of fish. Aquacult Eng 69, 7-17 (2015)

23. Zhou, C., Zhang, B., Lin, K. et al.: Near-infrared imaging to quantify the feeding behavior of fish in aquaculture. Comput Electron Agric 135, 233-241 (2017)

24. Downey, A.B. and Koutsopoulos, H.N.: Primitive-Based Classification of Pavement Cracking Images. J Transp Eng 119 (3), 402-418 (1993)

25. Beucher, S. and Lantuéjoul, C., Use of Watersheds in Contour Detection, International workshop on image processing, real-time edge and motion detection, RENNES,FRANCE, pp. 391-396. (1979)

26. Beucher, S., Watersheds of functions and picture segmentation, IEEE International Conference on Acoustics, Speech, and Signal Processing, 1982, IEEE, Paris, France, pp. 1928-1931. (1982)

27. Moreda, G.P., Muñoz, M.A., Ruiz-Altisent, M. et al.: Shape determination of horticultural produce using two-dimensional computer vision – A review. J Food Eng 108 (2), 245-261 (2012)

28. Brassel, K.E. and Reif, D.: A Procedure to Generate Thiessen Polygons. Geographical
Analysis 11 (3), 289-303 (1979)

29. Thiessen, A.H.: Precipitation averages for large areas. Monthly Weather Review 39 (7), 1082-1084 (1911)