2019 年度情報処理学会関西支部 支部大会

D-102

A proposal for a new method of fish species and size prediction byrecognizing fishing vibration pattern using machine learning
Shuichi Fukuda†

1.

Seong Eon Hong†
Hanbit Oh†
Yuki Matsuda†
Keiichi Yasumoto†

Introduction

In recent years, a fishing as a sport has become popular accompanied with developments of fishing gadgets.
The gadgets combined with cutting-edge technologies
keep being designed and improved. For example, somefishing products have an ability of sensing an impact [1]when a fish touches a rod and when the nod recognizes
the impact it notifies by using a led light which is attached on the rod. Many researchers and developers
are trying to come up with a more useful fishing equipment.
In this paper, we will design and develop a new technology for developing a new fishing device. Our aim
is to investigate if we can develop a device which finds
out a size and species information of a fish when thefish bit the rod. This will be realized by using a machine learning analysis with a movement of the rod andfeatures of information given by a fish.
The contents of this paper is structured as follows. In
Section 2, the existing studies related to the proposed
work will be presented. In section 3, features of fishes
when they swim will be introduced based on the result
acquired by experiments. In section 4, the method torealize our goal will be described. The plans for the future experiment will be covered in section 5 and finallya conclusion is given in section 6.

2.

Related work

Many studies have been carried out on recognition of
species and size of fishes so far. Strachan et al. tried toacquire these kinds of information by using a patternrecognition method [2]. They investigated feasibility of
their method based on the data bank of pictures thatthey collected for the analysis. Although the approachis diﬀerent from our work since they used a static data
for the recognition while dynamic data are analyzed in
our work, they could distinguish the type of fishes witha relatively high accuracy. It gives an intuition thatstatic data could be applied to distinguish species offishes.
†Nara Institute of Science and Technology

Hyuckjin Choi†

White et al. used a computer vision technique toidentify a fish [3]. They used the dynamic data collected from fishes passing a conveyor underneath a digital camera. Although they only succeeded to distinguish whether the fish is a round type or a flat type,it was meaningful since they used the dynamic data torecognize a fish.
Randy and Johnson have patented light-emitting fishing float [4]. They attached the sensor of lighting on ahook of a rod where the switch responses when a fishcontacts a hook.
As mentioned above, there are many existing studies
to identify fish in the context of fishing, however, tothe best of our knowledge, there was no research paper
dealing with a method to find the fish’s species by using
a rod. Therefore, in this work, we try to implement a
sensor device which can be attached to a fishing rod to
capture information of a fish, and identify the size andspecies of the fishes through analysis of the captured
data through the rod taking into account the traits offishes.

3.

Features of fish swimming

Some swimming features of fishes are created depending on the diﬀerences in vibration and the movement of a rod. Those features of swimming may help
predict the fish species and its size. The fish swimming traits which are related to our research will be
described as follows.
Firstly, various swimming forms of fishes are described.
The swimming forms are categorized by which parts ofthe body (fin) the fishes use and the way of swimming
using those parts.
Secondly, the fast-start, a brief behavior which is asudden acceleration caused by reaction, is discussed.
The report by Beamish [5] defined three major categories of swimming activities: sustained (over 200 min),prolonged (20s to 200min) and burst (under 20s). In
our case, the fast-starts are included in burst swimming. Because we are focusing on burst swimming, theother swimming activities are excluded from our target
swimming styles.

Swimming direction(A)Rajiform(C) Labriform(B) Diodontiform

Head movement

Head movement Head movement

Head movement Little head movement(D)Gymnotiform(F) Tetraodontiform(G) Balistiform(E) Amiiform(e) Ostraciform(c)Thunniform (d)(b)

Carangiform(a) Anguilliform (b) Sub-carangiform (c) Carangiform (d) Thunniform(a)
Subcarangiform

Eels

Salmonids

MackerelTunas(e) Ostraciiform
box fishBox

Anguilliform

Fig. 2: Caudal trunk swimmers

There are a varieties of fish swimming styles. ESI(Environmental Science Investigation) [6] divided fishes
into twelve groups based on the percentage of usedbody parts which are calculated based on movementin swimming (Fig. 1).
On the other hand, Sfakiotakis [7] reviewed some paper regarding fish swimming modes. According to thepaper, some fishes move their body and caudal fin to
generate propulsion. Some other species move forward
by using their median and paired fins movement. The
fishes involved in the former group swim faster so thatthey can easily hunt or escape from a predator. However, they are not able to turn around rapidly, resulting
in a diﬃculty of hiding from a predator. In the caseof latter group, on the contrary, they swim relatively
slowly but they can make a rapid change of direction.
Thus, they can live in coral reefs.
We describe the details of those swimming styles asfollows. The twelve groups shown in Fig. 1 are categorized into two groups. One group is using medianand paired fin while the other group is using body andcaudal fin.

Amiiform, (F) Tetraodontiform and (G) Balistiform,
are formed of one group based on its body parts which
are using the median and paired fin for swimming (Fig. 1).
The first one is Rajiform (Fig. 1-(A)) locomotion
which uses horizontal pectoral fins and moves like awave. For example, rays, especially mantas use thissort of locomotion. Secondly, Diodontiform (Fig. 1(B)) locomotion also uses the pectoral fins, however, it
moves vertically and a porcupine fish is one example ofthis locomotion. Thirdly, Labriform (Fig. 1-(C)) locomotion also uses vertical pectoral fins but it is rather
stiﬀer than the former groups. Therefore, they drag thefins through the water like a rowing motion for swimming. Fourthly, Gymnotiform (Fig. 1-(D)) fishes use
undulations of a long anal fin and a knife fish shows
this kind of motion. Fifthly, Amiiform (Fig. 2-(E))
locomotion uses a long dorsal fin which can move undulate for swim and Bowfin is included in this motion.
Sixthly, Tetraodontiform (Fig. 1-(F)) uses anal fins anddorsal to swim. They flapped their anal fins and dorsal
simultaneously. Lastly, Balistiform (Fig. 1-(G)) alsouses both anal and dorsal fins. The diﬀerences in median and paired fin propulsion swimming styles resultin diﬀerences in the frequency of a fin vibration, andthe magnitude of force of fin movements. It can also
be extracted as one of features made from vibration of
the rod.

3.1.1

3.1.2

Fig. 1: Fish swimming style [6]3.1

Fish swimming style

Propulsion using Median and Paired Fin

Targeting seven fishes which are (A) Rajiform, (B)Diodontiform, (C) Labriform, (D) Gymnotiform, (E)Propulsion using Body and Caudal Fin

Targeting five fishes, which are (a) Anguilliform, (b)Sub-carangiform, (c) Carangiform, (d) Thunniform and(e) Ostraciiform are involved in one group based on its
body part such as a body in general and a caudal fin(Fig. 1).
First of all, an Anguilliform (Fig. 1-(a)) fish has along and thin body such as eels. They move like awave as it utilizes the entire body called undulation.
Therefore, the range of their head movement is significantly wide compared to the length of their body(Fig. 2-(a)). Secondly, a Sub-carangiform (Fig. 1-(b))
fish also swims like a wave as well, but their body is
stiﬀer than Anguilliform fish in general (Fig. 2-(b)).
So they are able to swim faster than the former groupbut they have low mobility (such as salmonids) (Fig. 2(b)). Thirdly, a Carangiform (Fig. 1-(c)) fish is stiﬀer
and they can swim faster than the former groups. Their
movement is concentrated on the caudal and fin. Thus,
they rapidly oscillate their tail like a fan when they are
swimming. For instance, a Mackerel uses the carangiform (Fig. 2-(c)). Fourthly, a Thunniform (Fig. 1-(d))
fish has a large and crescent-shaped tail. They rapidly
oscillate the tail with strong peduncle which connects
the body and tail (such as tunas and a shark) (Fig. 2(d)). Therefore, they can make a high speed and longdistance swim which enables them to chase a prey andrun away from a predator. Lastly, an Ostraciiform(Fig. 1-(e)) fish only uses the fin oscillating for swimming. Therefore, the range of their head movement
is significantly small compared to the former groups(Fig. 2-(e)). For instance, a boxfish uses the ostraciiform.
3.2

Fast-start

Fast-start is a brief and sudden acceleration made byfish. This action is very important for fishes when encountering a predator or prey. Domenici [8] analyzed
those fast-start actions by the kinematics and performance. Fast-start determines the outcome of a predator, prey interactions in terms of feeding success or survival. Weihs [9] divides fast-starts into three kinematicstages (Fig. 3):stage 1 : the preparatory stroke
stage 2 : the propulsive stroke
stage 3 : continuous swimming or coasting
These three stages will be of great help to determine
the range of extracting features.

Stage 1

Stage 2(the preparatory stroke) (the propulsive stroke)

Stage 3
(continuous swimming or coasting)Fig. 3: Sequence of the fast-starts

4.

Methodology

The method proposed in this paper, the fish species
and its size are going to be predicted by a 9-axis sensor using machine learning. Firstly, we describe about
the 9-axis sensor which will be attached to the fishingrod. Secondly, we discuss about two types of features
extracted from sensor data. One is time-based extraction and the other is a frequency-based extraction. Finally, we will describe the details of machine learning
approach regarding how to predict the fish species andits size.
4.1

SensorIn our proposed method, we use two sensors. We use
a small sensor board called Senstick [10, 11, 12, 13] andattach it on the tip of fishing rod. SenStick is tiny andlight (3g including a battery) and can measure the 9axis acceleration and communicate with a smartphonethrough BLE. The attached position of Senstick is a tipof a rod since the tips are the most fluctuating pointof the fishing rod. The sensitivity of Senstick can be
set to 1G, 2G, 4G and 8G. Therefore, it is adaptableto various situations.
Moreover, the fishing vibration data will be obtainedusing a smart device which is attached near the fishing rod. The type of the smart device is iPod touch
manufactured by Apple Inc. [14]. It has an acceleration sensor, gyro sensor and compass. The attachment
position of both devices are shown in Fig. 4.
Sampling frequency of iOS App running on iPod
touch and Senstick is 100 Hz. This frequency was determined by our preliminary experiment that catches
two Hera-huna fishes. One is length of 350mm and theother one is 280mm. In this experiment, we obtained
the vibration data from sensors observing acceleration

on the iOS app with sampling frequency of 10 Hz. With
this sampling rate, however, it was not enough to visualize the frequency spectrum of fish vibrations because
the maximum resolution of FFT is only 5 Hz. Thus,
it is diﬃcult to compare the diﬀerences of fish species
and its size by the fish vibration frequency only.

Senstick

iPod touch

Fig. 4: The position of sensors

4.2

Features extraction

As we described the fish swimming styles, there areseveral patterns which are related to the fish species
and the shape of fish. Therefore we will predict the fish
species and its size by using those patterns with our feature extraction method. In the plan of our feature extraction, there are two types of features extracted fromacceleration data. One is time-based extraction andthe other is frequency-based extraction. Time-based
extraction set the first 10 seconds time-series sensor
data to the target.
Time-based extraction is calculated by using 10 seconds time-series data with the statistical function suchas average, standard deviation, maximum and so on.
These features are able to describe how the fishes fightwith the rod and the line and how strong their strengthis when they pull the line. We considered that thefish size and weight aﬀect the vibration amplitude andthe turning reaction of fishes because a big fish swims
stronger than a small fish while the big fish is harder
to turn around compared to tiny fishes.
In frequency-based extraction, we use the FFT (Fast
Fourier Transform) [15] for the analysis of fish vibration
frequency. The FFT converts a time series of equally

separated values from the discrete time domain to thediscrete frequency domain. Vibration frequency data
includes the information of fish’s swimming style. We
consider that fish vibration consists of fish’s head movement, body and caudal fin propulsion. Generally, user’s
rod movement is also considered as noise.
4.3

Model building

The purpose of our research is to predict the fish
species and its size with machine learning model. The
flow of model building is shown in Fig. 5. Firstly, it
starts at feature extraction from the vibration data.
Secondly, we concentrate on sorting features data andcollect ground-truth labels (fish species and size) toprepare training data for machine learning. Thirdly,
two models are built on the data collected. One is
the model which predicts the fish species. The othermodel is to predict the fish size. Lastly, the test will bedone by intentionally inputting features data withoutlabels and see if the fish species and size are predicted
correctly by the learned model.
Several machine learning models will be used to compare prediction results by the models. In this case,machine learning methods such as supervised learningmodels, Decision Tree, Random Forest and SVM (Support Vector Machine) will be implemented.
Decision Tree [16, 17] is a graph that uses a branching method to illustrate every possible outcome of adecision. Random Forest [18, 19], which is one of thestrong machine learning models. Random forest consists of a large number of individual decision trees that
operate as an ensemble. Each individual tree in the
random forest results in a class prediction and the classwhich is the most appeared becomes the model’s prediction result. Therefore, Random Forest can learn thevarious types of features based on many trees. SVM[20] is a supervised machine learning algorithm which
is used for classification or regression problems. It uses
a technique called the kernel trick to transform data.
And then it utilizes these transformations to find an
optimal boundary between the possible outputs.

5.

Experiment plans

In order to collect various species of fish data, a lotof fishing spots will be visited with the sensing devices
such as the Sensticks and the smart device. To minimize the eﬀect of waves, choice for a non-floating fishing spot is desired. Also, when the fish bites the rod,the line between the tip of the rod and the fish will

Conger eel
24cm

Red tilefis h
24cm

Red tilefis h
24cm

Red tilefis h
24cm

Red tilefis h
24cm

Red tilefis h
24cm

Red tilefis h
24cm

Training dataset
Model building

Movement of nod

Input

Predict

1

Red tilefish

Vibration signalof smartphone

MachineLearning
model

2

35cm

Fig. 5: The procedure of model building

ing induces vibrations and result in the fish doing faststart. So, it should be considered regarding the method
to minimize the noises. This would be resolved by installing a force sensor or acceleration sensor in the reachof a human hand and observing the noises from humanmotion. Finally, we expect this research paper to contribute to a fishing community by sharing the refined
data which is collected for the experiment. Furthermore, it is expected to have positive eﬀects by helping
people collect information of fish individually.

Acknowledgment
This work was supported by CICP (Creative andInter-national Competitiveness Project) of NAIST.

References
be stretched strongly and straight. At that time, thecondition is not including the wave vibration because
the line is straight.
Our fishing style is “single-hook fishing” which can
hit only one fish at one time. At least five species fishdata will be collected and more than ten fish will beneeded for one species. Thus, the total number of fishdata is expected to be more than fifty.
In evaluation of machine learning model, the features
such as accuracy, precision, recall, and F-measures areused as numeric indexes. Also, to review the predicted
details, we calculate a confusion matrix. Confusion
matrix is often used to describe the performance of aclassification model on a set of test data that the true
values are already known. Evaluation method is 5-holdcross validation [21].

6.

Conclusion

In this paper, we proposed a method for predicting
the species and the sized of fishes from the vibration of
the fishing rod. To capture the vibrations and movements transmitted from the fishing rod, we attach a
Senstick and a smart device to specific positions of arod.
There is a potential problem in the proposed methodthat the vibrations and movements which are delivered through the fishing rod are likely to contain a
bit of noise by many factors such as waves, currents,
and bumps of obstacles. Moreover, there are undesired
noises which are made by a fisher such as an unnecessary touch to the rod and shakes generated when thefisher hook a fish. The noises generated from hook-[1] Gary W Krutz and James W Kruse. Force detecting and indicating apparatus for fishing rods,
September 15 1987. US Patent 4,693,125.
[2] Norval James Colin Strachan, Paul Nesvadba, andAlastair R Allen. Fish species recognition by
shape analysis of images. Pattern Recognition,
23(5):539–544, 1990.
[3] Darren J White, C Svellingen, and Norval JC Strachan. Automated measurement of species andlength of fish by computer vision. Fisheries Research, 80(2-3):203–210, 2006.
[4] Randy D Johnson and John F Johnson. Highintensity light-emitting diode fishing float, November 2 1999. US Patent 5,974,721.
[5] FWH Beamish. Swimming capacity. Fish physiology, 7:101–187, 1978.
[6] Vanishing fish :Swimming styles among
fish.
available from https://esi.stanford.
edu/exercise/exercisefurther2.htm (accessed
2019-07-10).
[7] Michael Sfakiotakis, David M Lane, and J Bruce C
Davies. Review of fish swimming modes foraquatic locomotion. IEEE Journal of oceanic engineering, 24(2):237–252, 1999.
[8] Paolo Domenici and R Blake. The kinematics andperformance of fish fast-start swimming. Journalof Experimental Biology, 200(8):1165–1178, 1997.
[9] D Weihs. The mechanism of rapid starting of slender fish. Biorheology, 10(3):343–350, 1973.

[10] Yugo Nakamura, Yutaka Arakawa, Takuya Kanehira, Masashi Fujiwara, and Keiichi Yasumoto.
Senstick: Comprehensive sensing platform with anultra tiny all-in-one sensor board for iot research.
Journal of Sensors, 2017, 2017.
[11] Yutaka Arakawa. Senstick: sensorize every things.
In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM
International Symposium on Wearable Computers,
pages 349–352. ACM, 2015.
[12] Yugo Nakamura, Yutaka Arakawa, and Keiichi Yasumoto. Senstick: a rapid prototyping platform
for sensorizing things. In 2016 Ninth International
Conference on Mobile Computing and Ubiquitous
Networking (ICMU), pages 1–6. IEEE, 2016.
[13] Yugo Nakamura, Yutaka Arakawa, Takuya Kanehira, and Keiichi Yasumoto. Senstick 2: Ultratiny all-in-one sensor with wireless charging. In
Proceedings of the 2016 ACM International Joint
Conference on Pervasive and Ubiquitous Computing: Adjunct, pages 337–340. ACM, 2016.
[14] Apple Inc. : iPod touch. available from https://www.apple.com/jp/ipod-touch/ (accessed 201907-20).
[15] William T Cochran, James W Cooley, David L
Favin, Howard D Helms, Reginald A Kaenel,
William W Lang, George C Maling, David E Nelson, Charles M Rader, and Peter D Welch. Whatis the fast fourier transform? Proceedings of theIEEE, 55(10):1664–1674, 1967.
[16] Yoav Freund and Llew Mason. The alternating decision tree learning algorithm. In icml, volume 99,
pages 124–133, 1999.
[17] S Rasoul Safavian and David Landgrebe. A survey of decision tree classifier methodology. IEEE
transactions on systems, man, and cybernetics,
21(3):660–674, 1991.
[18] Andy Liaw, Matthew Wiener, et al. Classification
and regression by randomforest. R news, 2(3):18–
22, 2002.
[19] Mohammed Khalilia, Sounak Chakraborty, andMihail Popescu. Predicting disease risks fromhighly imbalanced data using random forest.

BMC medical informatics and decision making,11(1):51, 2011.
[20] Johan AK Suykens and Joos Vandewalle. Leastsquares support vector machine classifiers. Neuralprocessing letters, 9(3):293–300, 1999.
[21] Tadayoshi Fushiki. Estimation of prediction error by using k-fold cross-validation. Statistics andComputing, 21(2):137–146, 2011.