# Remote Context Monitoring of Actions and Behaviors in a Location Through the Usage of 3D Visualization in Real-Time

John Conomikes, Zachary Pacheco, Salvador Barrera, Juan Antonio Cantu, Lucy Beatriz Gomez, Christian de Los Reyes, Juan Manuel Mendez Villarreal, Takao Shime, Yuki Kamiya, Hideki Kawai, et al.

► To cite this version:

John Conomikes, Zachary Pacheco, Salvador Barrera, Juan Antonio Cantu, Lucy Beatriz Gomez, et al.. Remote Context Monitoring of Actions and Behaviors in a Location Through the Usage of 3D Visualization in Real-Time. 9th International Conference on Entertainment Computing (ICEC), Sep 2010, Seoul, South Korea. pp.476-478, 10.1007/978-3-642-15399-0_64 . hal-01059018

HAL Id: hal-01059018

https://inria.hal.science/hal-01059018

Submitted on 29 Aug 2014

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Remote Context Monitoring of Actions and Behaviors in
a Location Through the Usage of 3D Visualization in
Real-Time

John Conomikes¹, Zachary Pacheco¹, Dr. Salvador Barrera², Juan Antonio Cantu²,
Lucy Beatriz Gomez², Christian de los Reyes², Juan Manuel Mendez Villarreal²
in collaboration with

Takao Shime³, Dr. Yuki Kamiya³, Hideki Kawai³, Dr. Kazuo Kunieda³,
Dr. Keiji Yamada³

¹ Carnegie Mellon University, Entertainment Technology Center (ETC), 800 Technology Drive, Pittsburgh, PA, 15219, USA

²Universidad de Monterrey (UDEM), Engineering and Technology Division, Av. Morones Prieto 4500 Pte. San Pedro Garza Garcia, C. P. 66238, N. L. Mexico

³NEC C&C Innovation Research Laboratories, 8916-47, Takayama-Cho, Ikoma, Nara 630-0101, Japan

{jconomik, zpacheco}@andrew.cmu.edu, {sbarrer1, jcantaya, lgomez20, xpiotiav, jmndezvi}@udem.net, t-shime@ce.jp.nec.com, y-kamiya@fn.jp.nec.com, h-kawai@ab.jp.nec.com, k-kunieda@ak.jp.nec.com, kg-yamada@cp.jp.nec.com

**Abstract.** "Remote Context Monitoring of Actions and Behavior in a Location Through the Usage of a 3D Visualization in Real-time" is a software application designed to read large amounts of data from a database and use that data to recreate the context that events occurred to improve understanding of the data.

**Keywords:** Multi3D, Visualization, Remote, Monitoring, Panda3D, Real-Time

# 1. Introduction

This prototype is the result of a long project development made at the Entertainment Technology Center where work was done in conjunction with NEC and the Universidad de Monterrey.

The goal of this project is to present huge amounts of data, not parse-able by a single person and present it in an interactive 3D recreation of the events that the sensors detected using a 3D rendering engine known as Panda3D.

While there is a lot of work in this field one of the unique angles of this project is the type of data is designed to build the recreation from. This data is from NEC's
LifeLog system which tracks a wide variety of detailed information on what each employee in the monitored space does daily on a second to second basis.

Additionally, the data can be viewed from anywhere in the world, not just the monitored laboratory.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Initial 3D shaded model for the Southern laboratory.</figcaption></figure>

## 3. System Architecture

Our entire system is built on NEC's LifeLog system which is responsible for gathering the large amount of data that is needed for the software to operate. See Figure 4 below for a view of the ceiling with installed sensors.

<figure><img src="image_3.png" /><figcaption>Fig. 4. Ceiling of the South Laboratory with installed sensors.</figcaption></figure>

Employee location is detected through the use of IR emitters on employees and receivers mounted on the ceiling, though approximately 25% of all location data is "8022" which is the code for a person who is not detected by any IR receiver on the premises. Ambient sound level data is collected by over 90 microphones installed in the ceiling.
All E-mails send to or from monitored employees are also stored, though addressees that are not monitored are stored only as "Company Employee" or "Recipient Outside Company".

Additionally, extensive information is pulled from the computer operations of each monitored employee. Statistics such as key presses, mouse clicks and mouse movements in the past second. Further, they track the currently active process running on the computer and the most recently accessed file. Even all of the currently running processes in the background. Finally they log all of the employee's internet access, though this last piece of information can be disabled by the employee. Finally, each employee has a wireless button that they carry with them that records when it was pressed and if pressed for more than one second, it also reports the duration of the press.

Also, while not related to people, 16 RFID readers are used to track the location of resources (e.g. books, laptops) which have RFID tags on them, as they move around the office. It also tracks which employee is using each particular resource.

The flow of information is quite simple, the LifeLog system polls the sensors for their latest information. It then takes this information, timestamps it and outputs it to a simplified YAML[1] format and stores this information on a server. Out program then connects to the server and requests the files required to view the time the user wishes to view, loads the needed information into memory in python data structures and displays the recreated events to the user.

Due to security restrictions at NEC, the data is only accessible locally or through a Virtual Private Network (VPN) connection. However, since the only remote action that is being performed with the software is reading data from the server, with less strict security measures, the software can function anywhere without the need for any special access permissions.

# 4. Experimental Results and Conclusion

While the system does use a large concentration of sensors in a small area and is generally very invasive, it does mean there are many promising opportunities for future research to improve on both the technology and software. While not ready for industry yet, with the inclusion of other research as well further improvement of the current software this seems to be a promising technology and may prove to be the next big step in combining multiple different information gathering technologies.

# References

1. B. Hibbard, "Visad: connecting people to computations and people to people," SIGGRAPH Computer. Graphics., vol. 32, no. 3, pp. 10–12, 1998.