Intelligence, Informatics and Infrastructure, Volume 5, Issue 1, 2024

Monitoring Human Activities in Riverine Space using
4K Camera Images with YOLOv8 and LLaVA:
A Case Study from Ichinoarate in the Asahi River
Shijun PAN1, Keisuke YOSHIDA2, Yuki YAMADA3, Takashi KOJIMA4
1Member of JSCE, Doctoral Student, Graduate School of Environmental and Life Science, Okayama University(Tsushima-naka 3-1-1, Kita-ku, Okayama 700-8530, Japan)
E-mail: p4b36znn@s.okayama-u.ac.jp (Corresponding Author)
2Member of JSCE, Associate Professor, Graduate School of Environmental and Life Science, Okayama University
3Member of JSCE, Master Student, Graduate School of Environmental and Life Science, Okayama University
4Senior Researcher, TOKEN C. E. E. Consultants Co., LtdA uniform nationwide survey on riverine space utilization has been conducted approximately every fiveyears as part of the "Census of Rivers and Waterfront Areas" in Japan, for properly promoting river projects
and river management. Considering significant effort required for human tasks, the survey is commonly
carried out for seven days per year. Then, the present river situation is estimated roughly through the year,
based on the limited survey results. Therefore, it is challenging to grasp the actual conditions on weekdays,
holidays, and at different times of the day. Accordingly, it is difficult to examine the effect of individual
river maintenance work quantitatively over years. For this study, the authors in this research tried torecognize human activities on the river bank automatically from 4K camera images taken near the Asahi
River diversion weir in Okayama Prefecture, using the object detection model YOLO (i.e., You Look OnlyOnce) with the large-scale multimodal model LLaVA (i.e., Large Language-and-Vision Assistant). Results
showed that the combination of these models has the potential to collect information on not only the numberand location of people but also various human activities, such as walking, running, and skateboarding.
Key Words: LMM, YOLOv8, Human Activity Estimation, River Monitoring, 4K Camera

necessitates a more detailed breakdown to precisely
understand river space utilization for the futureriparian-environmental improvement, maintenanceand citizen-convenience and protection (e.g. dogwalking-, kids-safety-based monitoring,), posing achallenge to surveyors from previous human-eyebased monitoring to automatic computeral analysis.
In recent years, many studies have been conducted
to detect people using AI technology. As an example,a study3) using the object detection model YOLO4) tounderstand the status of walking and waiting space
usage near bus stops showed the possibility ofquantitatively understanding items such as user
attributes (gender and age), the number of bus riders,
and the number of people using waiting benches. On
the other hand, human activities such as "walking"and "sports (including cycling and running)," which
are items in the survey on river space use, were notdetected. In addition, no research has been conductedto obtain information on people by combining object

1. INTRODUCTION
As part of the Census of Rivers and Waterfront
Areas, two essential surveys, the Survey on theNumber of River Space Users and the River Report
Card, are conducted to comprehensively understandriver space utilization1), 2). The user count survey,
guided by the Census Manual for Rivers andWaterfront Areas, occurs seven times (i.e., crowded
days selected by river management administrators formonitoring the usage of the river) per year to capture
the distinct characteristics of river use. For
understanding
thelong-termwater-friendly
environment-based situation in the riparian area,
despite providing a holistic view of general riverusage, the current methodology faces challenges in
capturing weekday patterns due to the balance
required between survey frequency, time, and cost.
The existing broad categorization of use patterns

89Intelligence, Informatics and Infrastructure, Volume 5, Issue 1, 2024

current practical issues that need the personnel-based
monitoring for a long term, the combination of thementioned models-based non-personnel-monitoring
automatic analysis results is an available method.

2. STUDY SITEFig.1(a) highlights the position of the study site in
the Okayama Prefecture. In Fig.1(b), the location andthe camera-direction of the device have been
depicted. Fig.1(c) showed the 4K camera from thedrone view. Fig.1(b) performed that Ichinoarate
within the bifurcation area of the Asahi River. InFig.1(b), the role of Ichinoarate becomes apparent asa crucial overflow weir responsible for channelingflood flow from the Asahi River into the distributary
Hyakken River. This pivotal function underscores theimportance of Ichinoarate in managing water flow inthe lower reach of the Asahi River. To facilitate acomprehensive understanding, Fig.1(b) also provides
insight into the perspective of the 4K camera, asdepicted in Fig.1(c). This camera's positioning
becomes instrumental in capturing the river behavior,especially during flood events. A sample image of 4K
camera during normal water was shown in Fig.2.

Fig.1 Study site3. METHODS(1) Specifications of video-taken device
In this research, data collection was derived froman 4K camera (Hikvision, DS-7600 Series) as
performed in Fig.1 with the resolution of 3840 ×2160 px and 20 fps (i.e., frames per second), theseriver monitoring cameras were previously installed toprovide real-time information on river conditions
during floods from 3rd, Aug. to 13th, Dec. 2021.
During the non-flood seasons, the 4K cameras also
continue working for the data collection.

Fig.2 A image of the Ichinoarate at 8:24 AM on November 10,
2021 during normal water from 4K camera dataset.

detection models and large-scale multimodalsystems.
In response, this study proposes a detection
method leveraging YOLOv84) and LLaVA5) toaddress the limitations. Focused on riparian area, theinnovative approach aims to unravel citizen
behaviors using rivers. By merging the strengths of
YOLOv8 (i.e., person-based number counting) andLLaVA (i.e., person-based activities analysis), thestudy seeks to provide an efficient detection
approach, transcending current limitations that only
one model cannot overcome the tasks like, theperson-based number counting, activity analysis area
selection and offering an insight into river space
usage behaviors at the same time. Considering the(2) Outline of artificial intelligence models
In this research, there are two models that havebeen applied for the person-related number andactivities analysis from the computer-vision- and theLMM- based perspectives, individually. The
computer vision model, YOLOv8 (i.e., You Only
Look Once version 8) is a state-of-the-art deeplearning framework for object detection,
segmentation, pose estimation and classification
tasks, the YOLOv8 cannot yet identify the specificactivities without additional training. The multimodal model, LLaVA (i.e., Large Language and90

Intelligence, Informatics and Infrastructure, Volume 5, Issue 1, 2024

Table 1 Specifications of LLaVA and YOLOv8 applied in thisresearch.

Table 2 36 chosen cases for the analysis derived from multiple
situations (i.e., months, weeks, water situations and timeperiods)

further enhances the analytical depth, facilitating
analysis that unveils an understanding of the detectedactivities. By synergizing the capabilities ofYOLOv8 and LLaVA, capturing intricate details ofhuman interactions with heightened accuracy. As thefinal step in Fig.3 (d), the results are meticulously
compiled and processed, giving rise tocomprehensive reports and visual representations
that highlight identified human activities.

Fig.3 Flow chart of investigating human activities by using
YOLOv8 and LLaVA: a) 4K camera-based datacollection, b) Analyzing the person-based numbers using
YOLOv8 and LLaVA, c) Analyzing the person-based
activities using LLaVA, d) Practical analysis of theperson-based numbers and activities using LLaVA.

Vision Assistant) is a multi-modal model thatcombines vision and language for general-purpose
visual and language understanding. The parametersetting is written in Table 1.

4. RESULTS AND DISCUSSION
To optimize computational costs and refine theanalysis area, the authors strategically structured two
distinct groups for testing purposes. Group-A
specifically aimed to assess the detection capabilities
in full-screen and individual cropped images,
focusing on the varying locations concerning the 4K
camera, namely closer-distance and longer-distancelocations. This evaluation aimed to discern theeffectiveness of detection in different spatial(3) Work Flow of investigating human activities
In Fig.3 (a), the data collection phase begins bydeploying a 4K camera, strategically positioned to
meticulously capture the targeted area of interest.
Moving to the analysis stage, as depicted in Fig.3 (b)and (c), the YOLOv8 framework was for the object
detection, effectively locating various persons within
the recorded 4K footage. The integration of LLaVA

91

Intelligence, Informatics and Infrastructure, Volume 5, Issue 1, 2024

contexts, providing insights for the accurate personbased numbers.
Alternatively, Group-B was dedicated toinvestigating the impact of the analysis area's size on
confirming person-interacted activities. This groupdelved into understanding how adjusting the size ofthe analysis area influenced the outcomes,
particularly in scenarios involving interactions
among individuals. By categorizing the tests into
these two groups, the authors addressed locationrelated considerations in Group-A and size-related
considerations in Group-B, collectively contributing
to a more efficient and cost-effective analysis
process.
Images were selected by classifying cases in termsof weekday/holiday, morning/afternoon/night, andwater-effluence/normal-water, covering most riverconditions during the shooting period. As shown in
Table 2, 36 cases were selected as Group-C. A single
targeted image was chosen for each case, confirming
the presence of people on the river bank. Only inAugust, there were water effluence situations.

along the river bank in the upper frame. Notably,
when the cropping area is expanded, as demonstrated
in the Fig.6 downside, an additional element isintroduced: a woman with a leashed dog. Thisaugmentation in the cropping area induces adiscernible shift in the outcomes generated byLLaVA.
Drawing insights from the LLaVA-derived results,
it becomes apparent that when analyzing a group ofpersons, particularly when interactions among theseindividuals come into work, careful considerationneed to be given to the selection of the cropped area.
The behaviors introduced by the inclusion of aleashed dog underscores the importance of
judiciously defining the spatial boundaries of theanalysis. As such, the implications of alterations in
the cropping area on the interpretability of the resultsmust be taken into account, ensuring a nuanced andcomprehensive understanding of person-related
activities within a given scene.
(3) Group-C: Application on the practical cases.
To avert misclassification of activities resulting
from incomplete person coverage, a deliberate choice
was made to employ a cropping process depicted in
Fig.7. In consideration of the diverse periods
encompassing study site, the authors judiciously
selected 36 cases, meticulously detailed in Table 2,
derived from various situations. In consonance withthe findings from Group-A, the focus was placed oncloser-distance areas.
This methodology ensures that the images
encapsulate the entirety of interacting individuals,
thereby enhancing the precision of LLaVA's analysis.
Subsequently, the requisite analysis from theimages was methodically extracted, as illustrated inFig.8, denoted by the delineated yellow areas (theBounding-Box- and Mask-based results on theindividual objects shown in Fig.9). In Fig.10, theauthors applied the prompt in detecting the personbased number and activities. The outcomes ofperson-related activities and numbers across different
cases, as discerned through LLaVA, weresystematically compiled in Table 3. Half of theperson-related activities and numbers wereaccurately detected across all 36 cases comparing
with GT, that were counted by the author without
twice counting. From the output derived from theTable 3, the prompt is not totally accurate for the
person-based activities and numbers. In the future,
the other prompts are in the need for the accurate
detection.

(1) Group-A: Analysis on person in full-screen
images.
In Fig.4, a comparative analysis of results obtained
from LLaVA and YOLOv8 reveals that both methods
area. However, LLaVA faces challenges in extracting
any information from the full-screen image of thelonger-distance area. In contrast, YOLOv8 exhibits
the capability to detect three persons in this moredistant region. As illustrated in Fig.5, astraightforward prompt was employed to verify
LLaVA's ability to count person-based numbers on
individual cropped images (i.e., upper-half anddownside-half parts of the original image). Evidently,
the longer-distance area also encounters limitations
in extracting this information using LLaVA (i.e.,
misclassified 3 persons to 2 persons).
The findings from Fig.4 were subsequently
revealed that the closer-distance output effectively
extracted person-related data using both YOLOv8
and LLaVA. Conversely, the longer-distance output
exhibited a divergence in performance, withYOLOv8 yielding partially correct results andLLaVA misclassifying person-related numbers
entirely.
(2) Group-B: Analysis on person-interactedactivities.
Fig.6 upper performed a woman that is running,Fig.6 middle presents a depiction of two individuals,
a woman and a child, engaged in walking or running

92

Intelligence, Informatics and Infrastructure, Volume 5, Issue 1, 2024

Fig.4 Results of detecting the person-related activities were derived from full-screen using LLaVA, YOLOv8x-seg. Closer- andLonger- distance outputs were extracted by authors derived from Full-screen results, individually. (i.e., red- and yellow-line
stand for the closer- and longer-distance result, separately)Fig.5 Results of detecting the person-related numbers derived from closer- and longer- distance images using LLaVA, YOLOv8xseg, individually. (i.e., red- and yellow-lines are same as in Fig.4)

93

Intelligence, Informatics and Infrastructure, Volume 5, Issue 1, 2024

Fig.6 Results of detecting the person-interacted activities derived from cropped images using LLaVA. It showed a problem of LLaVA,
when the cropped area changed, the activities also change.

Fig.7 Process of how to choose the cropped images for the LLaVA analysis: 1. Cropping the down-half of the whole image area (i.e.,Closer-distance area), 2. Using YOLOv8 to extract the necessary factors (i.e., person, bike, handbag, dog), 3. Collecting thedetailed positions for cropping the images derived from the YOLOv8-based results (i.e., x-min, y-min, x-max, y-max), 4.
Cropping the image based on the detailed positions.

94Intelligence, Informatics and Infrastructure, Volume 5, Issue 1, 2024

Fig.8 Table 2-based 36 chosen analysis area were selected using the yellow boxes derived from the process (i.e., W.E. and N.W.
stand for water effluence and normal water, individually)Fig.9 Results of B-Box (i.e., Bounding Box) and Mask-based YOLOv8 on multiple classes from 36 cases shown in Fig.8.

95Intelligence, Informatics and Infrastructure, Volume 5, Issue 1, 2024

Fig.10 Results of detecting the person-interacted activities using prompt-assisted LLaVA.
Table 3 Person-related activities and numbers detected in this research using the 4K camera from 36 cases (i.e., green color stands forthe correctly detected). GT stands for the meaning of ground truth that were judged by the authors about person-related
activities and numbers.

Considering that CCTV cameras are more widelyused in river monitoring in Japan than 4K cameras,
the authors performed similar validations by resizing
the video captured with a 4K camera from its originalresolution (3840 × 2160 px) to a standard CCTV
camera resolution (1920 × 1080 px) in Table 4. As aresult, no significant differences were found between
original and resizing datasets in the YOLO andLLaVA analyses. This implies that the earlier resultsof this study may be applicable in cases using CCTV
cameras installed in most Japanese rivers.

96

Intelligence, Informatics and Infrastructure, Volume 5, Issue 1, 2024

Table 4 Person-related activities and numbers detected in this research using images resized to CCTV resolution from 36 cases (i.e.,green color stands for the correctly detected). GT stands for the meaning of ground truth that were judged by the authors
about person-related activities and numbers.

Tourism for offering materials related to rivermanagement tasks and 4K video images along theAsahi River and Hyakken River.

5. CONCLUSION
The current study introduces an innovative
methodology that integrates YOLOv8 and LLaVA
for the daily observation of person-based activity inthe riverine space. Leveraging the implementation ofprompts, this approach successfully detected personrelated activities and numbers in 18 out of the 36
cases examined. This collaborative fusion of
YOLOv8 and LLaVA, augmented by the guiding
influence of prompts, marks a significant stride
towards enhancing the efficacy of understanding thelong-termwater-friendly
environment-based
situation in the riparian area.

REFERENCES
1)
2)

3)

4)
5)

https://openjicareport.jica.go.jp/pdf/1000047169_01.pdf
Tsuruta, M. and Nakamura, K.: Cultural ecosystem service
characteristics of rivers ascertained from national census onriver environments. Advances in river engineering, Vol. 27,pp.323-328, 2021.
Hironori, K., Masashi Y., Yusuke N., Toshiki H., Akihito
O., Aya U.: Research on analysis of public space usage
using object detection technology. Artificial Intelligenceand Data Science, Vol. 4, Issue 3, pp.83-89, 2023.
https://github.com/ultralytics/ultralytics
https://github.com/haotian-liu/LLaVA(Received January 15, 2024)(Accepted February 29, 2024)

ACKNOWLEDGMENT: The authors appreciate
the Chugoku Regional Development Bureau,
Ministry of Land, Infrastructure, Transport and97