Team Project: Tharsus
By Sergio Ruiz-Bonilla, Joaquín Armijo-Torres

Outline
1. Presentation of the projecta) Tharsusb) The Problemc) Data and tools
2. Methodologya) Background subtractionb) Object detection algorithm (YOLO)c) Path tracking
3. Conclusions

The Project

is a robotics company
Designing and Manufacturinghttps://www.tharsus.co.uk/

is a robotics company

https://www.tharsus.co.uk/

Production line: The life in the workshop
A

Unpackingmaterials

B

Grabing
tools andcomponents

C

Assembly ofmachines

Ideal Case:BABABBABCC

Production line: The life in the workshop
A
Real Case:

B

Unpackingmaterials

Grabing
tools andcomponents

ACCBB

CB
BA

B

Assembly ofmachines

Production line: The life in the workshop
AReal Case:

B

Unpackingmaterials

Grabing
tools andcomponents

AC
C

Is it possible to register how welldistributed is the workshop?

B
B

C
B
B
A

B

Assembly ofmachines

How to approach to the problem?
Track the movement of people in a large area:• Recording the activities of a person in the production line
• Identify the person• Repeat during several minutes and for different people
Examine the data to find ineffectively located spots:• A RaspberryPi camera is used to register the data
• Python, OpenCV and some object detection algorithms can be used todo the analysis
https://opencv.org/

Methodology

Use background subtraction in order to detectmovement
ii+1
Background
...

i+n

Compare toi+n+1

Use background subtraction in order to detectmovement

Use background subtraction in order to detectmovementPlot Intensity vs time to detect high activity
segments

Plot Intensity vs time to detect high activitysegments
Minimumduration

Threshold

YOLO (You Only Look Once):
Unified, Real-Time Object Detection
arXiv:1506.02640

Redmon et al. (2015)

YOLO (You Only Look Once):
Unified, Real-Time Object Detection
arXiv:1506.02640

100x faster thancompetitors

https://pjreddie.com/darknet/yolo/

YOLO (You Only Look Once):
Unified, Real-Time Object Detection
arXiv:1506.02640
Single regression analysis
Resize of the input image

Redmon et al. (2015)

Prediction of boundingboxes

YOLO (You Only Look Once):
Unified, Real-Time Object Detection
arXiv:1506.02640

Redmon et al. (2015)

YOLO (You Only Look Once):
Unified, Real-Time Object Detection
arXiv:1506.02640

Redmon et al. (2015)

Can we apply YOLO to this problem?
• Short answer: Yes. Nevertheless, limitations arepresent in the process (open problem).
• We apply YOLO to identify the individuals onevery frame.

• Workers path at the workshop can be tracked
on real time.

YOLO outputs• Once a 'Person' is detected, useful
information is saved.

• Pixel position (x,y), width and heightof YOLO output boxes, andconfidence of the YOLO class.
• Can we find all the tracks of one
individual person?

Path tracking: the simplest approach?

CNNCNNPath tracking: siamese network approach

CNNFully
connectedlayers
CNNPath tracking: siamese network approach

Path tracking: siamese network approachTraining and validation loss for the siamese
network approachTraining and validation loss for the siamese
network approach• The training and validation highly depends on
how the data sets are selected.
• 13,000 pictures labeled.
• The accuracy could be improved by labelingmore data and training on more epochs.

Conclusions

Pipeline
Background subtraction
Raw Video+YOLO
x, y, boxes
Path tracking
x, y, boxes
for eachindividualHighactivityframes

Summary and conclusions• Workers activities can be tracked by using a combination of background subtraction and an
object detection algorithm. However, we require an additional tool to identify all theindividuals in the workshop.
• In comparison to different approaches, as CNN and unsupervised learning, the siamese
network approach works better.
• Although the siamese network approach gives a good accuracy in both the training and thevalidation, more data is need to produce results with a higher fidelity and solve theproblem.