Estimation of beef marbling standard for live cattle using
multi-input convolutional neural networkwith ultrasound images
Toshiki Katayama, Hirohumi Kawada, Masashi Nishiyama, Yoshio IwaiaaTottori University, Tottori, Japan
ABSTRACT

There is substantial interest among livestock farmers in estimating the value of live beef cattle from ultrasoundimages. In this study, we aimed to clarify the knowledge employed for the estimation of beef marbling standard(BMS) by experts in visual inspection. To do this, we conducted a field survey, which revealed that the experts
observe ultrasound images containing various body parts of live cattle: in particular, the loin part, iliocostalispart, and shoulder clod part. To automatically estimate the BMS value, we designed a convolutional neuralnetwork architecture that incorporates the experts’ knowledge. We demonstrated that our multi-input neuralnetwork, with ultrasound images of various body parts, obtained a high accuracy in BMS estimation.
Keywords: Beef cattle, ultrasound images, beef marbling standard, deep learning

1. INTRODUCTION
There is a strong demand from consumers for tasty beef. The beef cattle of Japan have high oleic acid content,so Japanese beef has earned a high reputation among consumers. In particular, the cattle of Tottori Prefecture
are attractive for breeding purposes. It is important for livestock farmers to produce beef cattle with highvalue, and early estimation of the value can increase farmers’ profits. Therefore, there is substantial interest
among livestock farmers in knowing the value of beef cattle at an early stage, before shipment. In Japan, thebeef marbling standard (BMS)—defined by the Japan Meat Grading Association—is a well-known metric fordetermining the value of beef by visual inspection. BMS quantifies the degree of marbling in the loin part ofbeef cattle after slaughter, expressed as a number from 1 to 12; a larger number corresponds to a higher quality.
The loin part, also called the rib-eye part, chuck-eye roll part, or M. longissimus thoracis part, is one of the mostvaluable body parts of beef cattle.
At present, to estimate the BMS value from live beef cattle before shipment, experts view ultrasound images
such as those shown in Fig. 1. Usually, the ultrasound images are acquired so that the region corresponding to
the loin part is clearly visible. The experts assess the BMS value visually, using their experience and intuition.
Consequently, the accuracy of the value depends on the experts’ skill. This causes the problem that inexperienced
observers may estimate the BMS value with a low accuracy. Therefore, it is desirable to assist the observers asa means of bridging the gap between inexperienced observers and experts.
In recent years, techniques have been proposed for estimating the value of beef cattle from ultrasound imagesof live animals, using machine learning,1 deep learning,2 and statistical analysis.3, 4 The existing techniques are
based on the general knowledge that experts employ when observing the region corresponding to the loin partto estimate the BMS from ultrasound images of live beef cattle. This general knowledge is considered mostimportant in BMS estimation conducted by experts, as described in the handbook of livestock.5 However, we
believe that experts exploit other knowledge, in addition to general knowledge. The observation of the loin partis undoubtedly very important, but there are also many other body parts in beef cattle. We believe that experts
observe the various body parts for BMS estimation using their experience and skill. To clarify the knowledge
exploited for the observation of body parts of beef cattle, we conducted a field survey on the BMS estimation
performed by experts. This field survey revealed the following two types of knowledge:
Further author information: E-mail: nishiyama@tottori-u.ac.jp

BMS:10???

Live beef cattle
Ultrasound image
ExpertInexperienced observer
Figure 1. At present, experts visually assess the BMS value of live beef cattle, using their experience and intuition. They
determine the BMS value by observing the loin part from ultrasound images, based on general knowledge. Inexperienced
observers may estimate the BMS with a low accuracy.

Knowledge 1: The experts do not only observe the loin image; they also inspect the backbone image andshoulder clod image. We describe the detail of the important body parts contained in these images in
Section 2.2.
Knowledge 2: The experts do not observe the body parts contained in the upper region of the loin image, theupper region of the backbone image, or the upper region of the shoulder clod image.
Existing techniques1–4 have not fully explored the use of multiple body parts of beef cattle in ultrasoundimages for BMS estimation. For this reason, we designed a convolutional neural network (CNN) architecture for
BMS estimation that incorporates the experts’ knowledge revealed by our field survey.
Point 1. To inspect the loin image, backbone image, and shoulder clod image, we use a multi-input CNN, which
can simultaneously input these images containing various body parts.
Point 2. As a preprocessing step for the CNN input layer, we crop the upper region of these images, which aremostly unobserved by experts.
We investigated the accuracy of BMS estimation obtained using our multi-input CNN, for various body parts,
on an original dataset. In this paper, we also discuss how to find the combinations of body parts that are mostuseful for achieving a high accuracy in BMS estimation.

2. FIELD SURVEY
2.1 Procedure conducted by experts for BMS estimation
To explain the experts’ knowledge, we first describe the procedure that they use to estimate the BMS of live beef
cattle. This field survey was conducted at the Livestock Research Center of Tottori Prefecture on September
12, 2018. Ultrasound images were acquired from beef cattle (Japanese Black steers), as shown in Fig. 2(a). An
ultrasound imaging system (HS-2200, Honda Electronics Co., Ltd.) and a probe (HLV-4212M, Honda Electronics
Co., Ltd.) were used, as shown in Fig. 2(b). The BMS estimation was performed by two people: an operator,
who moved the probe of the ultrasound imaging system, and an expert, who viewed the ultrasound images ona monitor. The operator placed the probe on the body surface of a live animal and acquired ultrasound images
in real time using the system, as shown in Fig. 2(c). The expert viewed ultrasound images on the monitor toobserve the cattle body parts contained in the ultrasound images and to estimate the BMS, as shown in Fig. 2(d).
We describe the detailed procedures followed by the operator and expert below.
Initialization: Determine the initial position of the probe.
The operator identified the region between the sixth and seventh ribs on the body surface of the live animal
before acquiring ultrasound images. After identification, the operator placed and fixed the probe in close contact
with the skin corresponding to the region between the sixth and seventh ribs. Figure 3(a) shows the initial
position of the probe.

Probe(a) Beef cattle(b) Ultrasound imaging system(c) Operator(d) Expert

Figure 2. Setting of our field survey on the BMS estimation performed by an operator and an expert.
Initial position

Sixth rib

First position

Second position

Backbone

Loin(Longissimus thoracis)

Seventh rib

(a)

(b)

(c)

Third position

Shoulder clod(d)Figure 3. Examples of the stationary positions of the probe. The red circle in the upper image indicates the stationary
position on the surface of the animal’s body. The red circle in the lower image indicates the stationary position controlled
by the operator.

Step 1: Acquire the backbone image.
The operator moved the probe from the initial position to the first position. Figure 3(b) shows the first positionof the probe. The operator acquired the ultrasound image, and the expert checked the backbone part containedin the image. We call this image the backbone image. Figure 4(a) shows an example of the backbone image.
Step 2: Acquire the loin image.
The same procedure was repeated with the probe in the second position (Fig. 3(c)) to acquire a loin image(Fig. 4(b)).
Step 3: Acquire the shoulder clod image.
The same procedure was repeated with the probe in the third position (Fig. 3(d)) to acquire a shoulder clod
image (Fig. 4(c)).
We describe how the expert observed the ultrasound images using their knowledge in the next section.

2.2 Knowledge used by experts for BMS estimation
Before describing the knowledge revealed by our field survey, we first describe the general knowledge traditionallyused for BMS estimation. The general knowledge involves simply observing the loin part contained in ultrasoundimages. This is because the loin part is the most important body part that is used to determine the BMS value
after slaughter. The experts estimate the BMS value by checking the density of white grain points in theultrasound image of the loin part. As the value of BMS increases, the loin part tends to be densely covered withwhite grain points. However, white grain points in the loin part are often not clearly visible because they dependon the individual differences between cattle. Therefore, experts in BMS estimation also observe various otherbody parts using the procedure described in Section 2.1. We describe two types of knowledge (named Knowledge
1 and Knowledge 2) revealed by our field survey below.

Backbone image

Loin image

Shoulder clod image

(a)

(b)

(c)

Figure 4. Examples of the backbone image, loin image, and shoulder clod image.
Backbone image

Loin image

Backbone

Small
portion of
loin
(a)

Trapezius

Shoulder clod image
Subcutaneous
fat

Loin
(Longissimus thoracis)
Iliocostalis(b)Shoulderclod(c)Figure 5. Examples of the body parts of beef cattle contained in the ultrasound images.

In Knowledge 1, The backbone image, loin image, and shoulder clod image are acquired in the steps describedin Section 2.1. The experts estimate the BMS value by observing all the white grain points that appear in theseimages. We describe the details below:• The backbone image contains the backbone part and a small portion of the loin part. Figure 5(a) shows anexample of these body parts. The experts check the density of white grain points in the small portion of
the loin part and its surrounding part. As the BMS value increases, these parts tend to be densely covered
with white grain points. The backbone image has a lower priority than both the loin image and shoulder
clod image.
• The loin image contains the loin part, iliocostalis part, the trapezius part. Figure 5(b) shows an exampleof these body parts. The experts first check the density of white grain points in the loin part, in the samemanner as described above. Second, the experts check the density in the iliocostalis part. When the BMS
value is low, the iliocostalis part tends to have a larger segment of white mass and a smaller area of dense
grain points. The loin image has a higher priority than both the backbone image and shoulder clod image.
• The shoulder clod image contains the shoulder clod part and the subcutaneous fat part. Figure 5(c) shows
an example of these body parts. The experts check the density of white grain points in the shoulder clod
part and its surrounding part. As the BMS value increases, these parts tend to be densely covered withwhite grain points. The shoulder clod image has a lower priority than the loin image, but a higher priority
than the backbone image.
In Knowledge 2, We describe the details below:• In the backbone image, the backbone part is ignored when estimating the BMS value. It is only used
to determine the first position of the probe of the ultrasound imaging system in step 1, as described in
Section 2.1.
• In the loin image, the trapezius part is ignored when estimating the BMS value. Because the trapezius
part is located under the skin, it always appears in the loin image.
• In the shoulder clod image, the subcutaneous fat part is ignored when estimating the BMS value. Because
the subcutaneous fat part is located under the skin, it always appears in the shoulder clod image.

・・・

Loin image
Input

Convolutional layers
(a)

BMS

FC layer・・・

Backbone image

Loin image

BMS

Concatenate
Shoulder clod image
Input

Convolutional layers(b)

FC layer

Figure 6. (a) Network architecture based on general knowledge. (b) Network architecture based on Knowledge 1, revealed
by our field survey. Here, “FC” means “fully connected.”

3. DESIGN OF NETWORK ARCHITECTURE BASED ON EXPERT KNOWLEDGE
We have designed a CNN architecture that incorporates the knowledge of BMS experts. There are some existing
CNN-based methods that use ultrasound images of cattle for other purposes, such as the automatic checkingof the body condition of dairy cows6 and the automatic detection of follicles in cattle.7 Recently, a CNNbased method for checking the beef value of cattle has also been proposed.2 We first consider the design of asimple network architecture for BMS estimation using general knowledge, in which the experts observe only theloin image. Figure 6(a) shows the simple network architecture based on this general knowledge. Because thisarchitecture does not allow for the incorporation of the knowledge revealed by our field study, we describe how
to design our improved network architecture below.
To incorporate Knowledge 1, described in Section 2.2, into the network structure, we use a multi-input CNN.
We aim to simultaneously observe the backbone image, loin image, and shoulder clod image, because these are allobserved by the experts, using multi-input layers. After applying convolutional layers to each of the ultrasoundimages, the feature maps are concatenated and the BMS is estimated. Figure 6(b) shows the network architectureof the multi-input CNN that processes the backbone image, loin image, and shoulder clod image.
To incorporate Knowledge 2, described in Section 2.2, into the network structure, we use a preprocessing
step, in which the upper region of each ultrasound image is cropped. We aim to ignore the body parts containedin the upper region of the loin image, backbone image, and shoulder clod image because these regions are notobserved by the experts. It is difficult to perform the segmentation of these body parts automatically because
their appearance is not clear and their boundaries are ambiguous in the ultrasound images. Therefore, we simply
crop the upper regions of the ultrasound images—showing the backbone part, trapezius part, and subcutaneous
fat part—which are ignored by the experts when they estimate BMS. Figure 7 shows examples of cropping theupper regions of the ultrasound images in the preprocessing step of the multi-input CNN.

4. EXPERIMENTS
4.1 Dataset
We evaluated the accuracy of BMS estimation using two classes: one class with a BMS value of 12, and theother with a BMS value of 5 or less. BMS estimation is fundamentally a multi-class classification problem, but
we simplify it to a two-class classification problem. The reason for this simplification is that BMS estimation isa very difficult task. Figure 8 shows examples of ultrasound images of live beef cattle in each of the two classes:
those with a BMS value of 12, and those with a BMS value of 5 or less. When we observe the loin images, theBackbone
Subcutaneous image
fat

Loin
image

Shoulder clod
image

Removeupper regions

Backbone
image

Loin
image

Shoulder clod
image

Cropping as a preprocessing step
Figure 7. Examples of cropping the upper regions of the ultrasound images in the preprocessing step of the multi-input
CNN.

Animal 1 Animal 2 Animal 3 Animal 4 Animal 5… Animal 21

Backbone
image…

BMS valueof 5
Loin image
or less…

Shoulder clod
image…
Animal 22 Animal 23 Animal 24 Animal 25 Animal 26 … Animal 42

Backbone
image…

BMS value
Loin image
of 12

…

Shoulder clod
image

…

Figure 8. Examples of ultrasound images of live beef cattle with a BMS value of 12 (bottom) and a BMS value of 5 orless (top).

differences that correspond to the different BMS values are not obvious; this is because of the large variation in theappearances of individual images. The same is true of backbone images and shoulder clod images. Even though
the experts’ knowledge, described in Section 2.2, is used to decide the BMS values, inexperienced observers
cannot easily distinguish the two classes. As a first step toward helping inexperienced observers estimate BMS,
we conducted experiments to evaluate the accuracy of this two-class classification problem.
We used a dataset collected by the Livestock Research Center of Tottori Prefecture. Ultrasound images wereacquired from Japanese black steers. The period of data collection was from May 2017 to October 2018. The
total number of beef cattle was 42: 21 animals with a BMS value of 12 and a further 21 animals with a BMS valueof 5 or less. For each animal, 90 backbone images, 90 loin images, and 90 shoulder clod images were acquired.

4.2 Accuracy of BMS estimation using our network architecture
We evaluated the effectiveness of our network architecture using the expert knowledge revealed by our fieldsurvey. We compared the following experimental conditions:Accuracy (%)

90

70

50
Baseline

Comparison 1

Comparison 2

Ours

Accuracy (%)

Figure 9. Accuracy of BMS estimation using the baseline, comparison 1, comparison 2, and our method.

90

70
50

None
10
20
30
40
Cropped size: Height of cropped area at top of ultrasound image (%)
Figure 10. Accuracy of BMS estimation while changing the cropped size of the upper regions in ultrasound images.

Baseline: We used the CNN of Fig. 6(a), designed to use only general knowledge, whereby the experts observe
only the loin image. This network consisted of three convolutional layers and pooling layers.
Comparison 1: We used the multi-input CNN of Fig. 6(b), incorporating only Knowledge 1, whereby the experts observe the backbone image, loin image, and shoulder clod image. Each stream of the network architecture
consisted of three convolutional layers and pooling layers.
Comparison 2: We used the CNN with the preprocessing step, in which the upper region of the loin image wascropped, to incorporate only Knowledge 2. This network had the same architecture as that used in the baseline
method, with the addition of the cropping step.
Ours: We used the multi-input CNN with the preprocessing step for cropping, to incorporate both Knowledge
1 and Knowledge 2.
We used the cross-entropy loss function with 64 batches and 50 epochs. We applied 21-fold cross-validation
three times. Figure 9 shows the accuracy of BMS estimation using the baseline, comparison 1, comparison 2,
and our method. The figure shows that the accuracy of our method was superior to that of the others. We
believe that the expert knowledge revealed by our field survey contributed to the increase in the accuracy of
BMS estimation.

4.3 Accuracy while changing the cropped size of the upper regions
As described in Section 3, our method cropped the upper regions of the ultrasound images using the preprocessing
step based on Knowledge 1. We believe that the classification accuracy depends on the cropped size of the upper
regions because there are individual differences among beef cattle. Therefore, we evaluated how the accuracy
varied with changes to the cropped size of the regions. The cropped size (i.e., the height of the cropped area,
measured from the top of the ultrasound image) was varied in 10% increments. We used the same experimental
conditions as those described in Section 4.2, with the exception of the cropped size. Figure 10 shows the accuracyof BMS estimation in ultrasound images as the cropped size varied. The highest accuracy was achieved when
the top 30% of each image was cropped. We believe that the bottom 70% of each image contains the body parts
that are most informative for BMS estimation.

4.4 Effectiveness of the combination of body part images
In this study, we assumed that the backbone image, loin image, and shoulder clod image all contribute toincreasing the accuracy of BMS estimation. To determine which images improve the accuracy, we evaluated all
combinations of these images that are input to the network. Figure 11 shows the accuracy of BMS estimation
using the multi-input CNN for all combinations of the three body part images. The figure shows that theAccuracy (%)

90

70

50

Backbone

Loin Shoulder clod Backbone
and loin(a)Backbone and Loin andshoulder clod shoulder clod
(b)

All

(c)

Figure 11. Accuracy of BMS estimation using the multi-input CNN on the backbone, loin, and shoulder clod images. (a)One image was used. (b) Two images were used. (c) All three images were used.

accuracy was highest when all the images were used, but high accuracy was also achieved when only the loin andshoulder clod images were used. This indicates that the loin and shoulder clod images are the most informative
for BMS estimation.

5. CONCLUSIONS
We conducted a field survey of livestock farmers, which revealed the types of knowledge that experts use whenobserving various body parts of live cattle—in particular, the loin part, iliocostalis part, and shoulder clod
part—for visual estimation of BMS. To automatically estimate the BMS value, we designed a multi-input CNN
architecture that incorporates the knowledge about body parts. Experimental results show that the accuracyof BMS estimation using our multi-input CNN with images of various body parts is higher than the accuracy
using a baseline CNN, which exploits only general knowledge. In future work, we intend to develop a method
for multi-class BMS estimation, and extend our accuracy evaluation to a large dataset of images.

ACKNOWLEDGMENTS
We would like to thank Mr. H. Takatori, Mr. M. Murakami, and Mr. M. Ooba of the Livestock Research Centerof Tottori Prefecture for their cooperation in our field survey and data collection.

REFERENCES[1] Shahinfar, S., Al-Mamun, H. A., Park, B., Kim, S., and Gondro, C., “Prediction of marbling score and carcass
traits in korean hanwoo beef cattle using machine learning methods and synthetic minority oversamplingtechnique,” Meat Science 161, 107997 (2020).
[2] Kvam, J. and Kongsro, J., “In vivo prediction of intramuscular fat using ultrasound and deep learning,”
Computers and Electronics in Agriculture 142, 521 – 523 (2017).
[3] Kuchida, K., Tsuruta, S., Vleck, L. D. V., Suzuki, M., and Miyoshi, S., “Prediction method of beef marbling standard number using parameters obtained from image analysis for beef ribeye,” Animal Science
Journal 70(3), 107–112 (1999).
[4] Pabiou, T., Fikse, W., Cromie, A., Keane, M., Nasholm, A., and Berry, D., “Use of digital images to predict
carcass cut yields in cattle,” Livestock Science 137(1), 130 – 140 (2011).
[5] Abe, A., “Basic seminar on agricultural science, new edition: Basics of livestock breeding (japanese),” Rural
Culture Association Japan (2008).
[6] Yukun, S., Pengju, H., Yujie, W., Ziqi, C., Yang, L., Baisheng, D., Runze, L., and Yonggen, Z., “Automatic
monitoring system for individual dairy cows based on a deep learning framework that provides identification
via body parts and estimation of body condition score,” Journal of Dairy Science 102(11), 10140–10151(2019).
[7] Zeng, T. and Liu, J., “Automatic detection of follicle ultrasound images based on improved faster r-cnn,”
Journal of Physics: Conference Series 1187, 042112 (2019).