# Designing an Experiential Media System: A Mobile Augmented Reality System for Family Situated Documentaries

Yi-Fan Chen

<figure><img src="image_1.png" /></figure>

## To cite this version:

Yi-Fan Chen. Designing an Experiential Media System: A Mobile Augmented Reality System for Family Situated Documentaries. 19th IFIP Conference on Human-Computer Interaction (INTERACT), Aug 2023, York, United Kingdom. pp.113-122, 10.1007/978-3-031-61698-3_11. hal-05305346

HAL Id: hal-05305346

https://inria.hal.science/hal-05305346v1

Submitted on 9 Oct 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
<figure><img src="image_4.png" /></figure>

# Designing an Experiential Media System:
A Mobile Augmented Reality System for Family
Situated Documentaries

Yi-Fan Chen(✉)

Farmingdale State College, Farmingdale, NY 11735, USA

Yi-Fan.Chen@farmingdale.edu

**Abstract.** Mobile media users utilize their devices to document their everyday experiences at the location. Users often share those situated documentaries both online and offline within their network. Mobile augmented reality systems (MARS) also open opportunities for users to retrieve their situated documentaries at the location. The current design research aims to propose a mobile experiential media design, specifically a mobile augmented reality system, for users to retrieve their family situated documentaries when they are physically present at the location. “Your Notes, My Love” MARS is designed based on the family’s need to share love, information, memories, and knowledge. It serves as a user-generated situated documentary MARS for everyday life. The main functions of the design include mapping experiences for the user and mapping experiences created by the user. Results from five user tests demonstrate that the mobile experiential design is interesting and well-received for family situated documentaries. Users appreciate the notes created by family members for them, which makes the experiences more personal and intimate. They also find the map view design helpful because it simplifies the process of adding or accessing family situated documentaries at specific locations. Users suggest adding a family tree function to map out intergenerational memories.

**Keywords:** mobile augmented reality · user-generated content · family

## 1 Introduction

Traditional media, such as papers, photos, audio, and videos, is utilized by users to create, store, and consume information. On the other hand, interactive media users employ interactive media to create, store, share, and retrieve information. With the increasing prominence of 5G mobile technology, experiential media, a subset of interactive media, has captured the attention of researchers and designers. Experiential media, including augmented reality systems, represents the integration of computational media into everyday experiences [27]. Experiential media, especially for mobile experiential media, has provided new opportunities for their users to create, store, share, and retrieve information at different scales of time and location. The opportunities provided by the mobile experiential media to easily create, store, tag, distribute, and retrieve content have potential
to change the generated and collective knowledge within the social networks. In simple terms, mobile experiential media could create ubiquitous integrative hybrid online-onsite culture. To clarify the definition of experimental media, John Pavlik proposes a new model of experiential media based on six primary qualities of the digital environment. These qualities are as follows: 1) interactivity, 2) immersion, 3) multi-sensory presentation, 4) algorithmic and data-driven, 5) first-person perspective, and 6) a natural user interface. Furthermore, he argues that experiential media transform traditional media content, such as news, from storytelling to information experiencing [23].

Research and design studies on mobile augmented reality systems (MARS), which are considered important mobile experiential media, have focused on entertainment [26, 30], training [14, 21], news storytelling [11, 24], retail [5, 25], and tourism [13, 15]. The findings indicate that MARS enhances user experiences and provides an additional layer of information to users at specific locations. In a study by Hollerer and colleagues, MARS content is referred to as situated documentaries, which utilize experiential media to overlay digital content, typically geolocated, onto users' real-world experiences [11]. Many of these research findings also emphasize specific experiences for one-way communication purposes, such as entertainment, training, news reporting, retail, and travel, from producers to users. In 2017, developers were encouraged to utilize Google's ARCore and Apple's ARKit AR frameworks to create MARS experiences. Those MARS experiences could provide users the chance to display online information on an onsite background and then interact with it [20]. For example, Pokémon Go has transitioned MARS from being a niche technology to becoming mainstream, seamlessly integrating it into everyday activities for many users. Google Lens is also gaining prominence as it assists users in searching for information about what they see at a given location and situation. MARS is evolving into a popular technological trend.

Previous research aims to understand the motivations and perceptions of users towards MARS situated documentaries created by media producers, primarily for one-way communication purposes at the locations. Other studies have shown that location-based services may give rise to concerns regarding the relevance of information and personal data privacy. However, there is a notable gap in attention towards comprehending users' motivations and perceptions when creating personal MARS situated documentaries in everyday life. The creation of MARS situated documentaries by users for users could potentially alleviate privacy concerns, as many studies in mobile media indicate that users are inclined to share within their networks but exhibit more reserved behavior with outsiders [16, 17]. This study explores the potential application of MARS in the creation and consumption of user-generated situated documentaries within a family context.

# 2 Literature Review

## 2.1 From Passive to Active Users

From the Gutenberg Bible to mobile augmented reality content, media producers and media content rely on the foundation of storytelling [22, 23]. The rapid development of advanced digital media significantly transforms media productions and impacts mediated
storytelling [3]. Traditional media storytelling, due to its different affordances, often fol-
lows a linear structure with a beginning, middle, and end. Audiences are typically passive
consumers of media content, with limited interaction such as turning a page or changing
a TV channel. On the other hand, experiential media allows users to actively engage
with the content and participate in the story itself. Digital media users have evolved into
more active participants, experiencing stories firsthand rather than merely receiving the
content passively. Pavlik categorizes mediated storytelling via digital media into three
stages: (1) storytelling using words; (2) storytelling using words, sounds, and images;
and (3) storytelling through experiential media. He argues that experiential media will
primarily focus on providing an environment for experiencing stories, while other media
will continue to play a significant role in public storytelling [22]. The experiential sto-
ries created through experiential media have the potential to build empathy and actively
engage users in empowering narratives [23].

## 2.2 Mobile User-Generated Content

Mobile user-generated content is produced by users who willingly contribute media and information while on the move. Because of the ubiquitous mobile media, users who consume and create their content on their mobile devices have seen rapid growth. When users are on the go, they can utilize their mobile devices to capture, edit, annotate, and present their experiential stories in real-time and location-specific contexts. Research indicates that mobile user-generated content often occurs within communities to fulfill diverse user needs, including networking and self-promotion [8]. For example, Open-StreetMap is a collaborative project that allows users to create free editable geographic data. OpenStreetMap users are highly motivated to share their geographic knowledge and actively engage in community participation. Mapping social stories has been found to be an important reason why users create the content [9]. OpenStreetMap has social mapping communities, such as humanitarian OpenStreetMap Team, which aims to share local stories for both humanitarian relief responses and humanitarian purposes in general to help people in the area [10]. Another similar design, “Urban Tapestries”, aims at building community and raising awareness among localized groups of people. The users use their mobile devices to add new content as they go about everyday activities to the location. The mobile user-generated content includes local knowledge, memories, stories, sensed information, and experiences [2].

More and more social media are utilizing the geo-located function via mobile
devices to create geographically anchored content and communication. Mobile media has
emerged as a powerful tool for storytelling in everyday life. Koskinen argues that mobile
media users create mobile user-generated stories in three main versions: instrumental,
expressive, and mundane [12]. In the instrumental version, mobile user-generated stories
are used to share information with other users, check timetables, and communicate routes
[6]. The expressive version of mobile user-generated story enhances creativity as users
share their everyday experience with stories, drama, and songs to others [1]. According
to the mundane version, the mobile user-generated stories are created to maintain social
connections and serve gossiping purposes [18]. Studying mobile user-generated stories
can provide insights into human activities. In one study, researchers examined geotagged
tourist photos in Rome to understand the experiences of American and Italian tourists in
the city. They utilized Flickr photos as their research data because user-generated stories can provide additional context to a physical location [7]. Other studies have found that mobile user-generated stories, such as photos, are used for various purposes including joking, emotional communication, and artistic activities. These artistic activities involve editing images and constructing stories from them. Mobile media users in Austria and Finland share photos to communicate immediate events as well as to convey emotions [19]. Mobile users create mobile user-generated stories mainly for interaction purposes. There is a higher prevalence of photos depicting everyday mundane activities, such as meals, dirty plates in the sink, and home streets, which are created for mundane interactions rather than expressive interactions [12]. It seems to be lots of mobile user-generated stories that are created for sharing daily experience.

## 2.3 MARS Designs for User-Generated Stories

The design community is highly interested in exploring how user-generated stories can utilize MARS to enhance user experience to the location. An early MARS design focused on understanding the reactions of local news audiences to user-generated local news stories. The study finds that users agree MARS is an interesting and acceptable solution for the content creators. It is usefulness in creating and consuming local news stories [28]. SituAR is designed for users to create AR stories at tourist hotspots. The study finds that user-generated storytelling and gamification are important aspects in the design of MARS [29]. A MARS design entitled, See What I See, is to facilitate users' companionship with their social networks while they are traveling overseas for the first time by themselves. The MARS stories created by the user's friends and family members based on the users' interests with personal notes provide the sense of presence for the backpackers while they travel [4].

In summary, MARS is used to share location information by providing interactive and immersive experiences for users. Some MARS are currently used to tell stories for the location by media producers. However, user-generated MARS stories appear to be rare. Prior studies have reported that the motivations of mobile media users generated stories are to share information, create local stories and build the local communities. This study aims to design a MARS solution that fulfills users' needs by allowing them to map their own stories within their network.

# 3 The “Your Notes, My Love” Design

## 3.1 The Design

MARS storytelling by users, for users, has the potential to transform everyday activities into extraordinary experiences. Imagine attending a graduation ceremony for a family member at a college, where you can create situated documentaries and tag the new college graduate with emojis. Now envision the college graduate walking near the college later on, when suddenly a notification appears on his/her mobile device. It is the situated documentary that you created for him/her at that very location.

A “Your Notes, My Love” MARS is designed for users to map their family moments at specific locations. It is designed to be used in everyday life to record love, information,
memories, and knowledge. “Your Notes, My Love” allows users to document important family moments and share them exclusively with their family network. It can serve as a mobile family album on a personal map. The mobile device will record the time, location, and weather data. The situated documentary includes text, images, audio, videos, and emojis, all created by the user. The user can also tag other family members.

When family members are near the relevant family memories, their mobile devices will push a notification, prompting them to decide whether to use the MARS platform to retrieve the situated documentaries created specifically for them. After reviewing the situated documentaries, family members can react, comment, or add to the existing situated documentaries. The main functions of the “Your Notes, My Love” MARS design include mapping experiences for the user and mapping experiences created by the user (see Fig. 1).

When a user creates a note and tags individuals with their current emotions, the data is stored in the system for other family members to access. If other family members are near the note’s location, the system sends a notification to inform them that there is a note for them to retrieve. Users can determine their relationship with the note creator and the distance from the location and then decide whether they want to walk to it. Users also have the option to react to and comment on existing notes.

Both “mapping experiences for the user” and “mapping experiences created by the user” are displayed on a map view. Finally, the Portfolio will include all nearby notes with family members who are involved in it. More design details are displayed in the following session for UI design (Fig. 2).

<figure><img src="image_5.png" /><figcaption>Fig. 1. “Your Notes, My Love” MARS design concept</figcaption></figure>

Based on user behavior research findings [6, 18], the model of experiential media [23], and user feedback from the previous “Your Notes, My Love” MARS design concept diagram, this design research has advanced to create sample UI designs for testing. The objective is to help users better grasp the design concepts. These UIs have been crafted using Adobe XD, with the primary goal of collecting additional feedback and refining the user interface to more accurately align with user needs.
In Fig. 2, there are five distinct UI representations: The UI #1 offers a simplified interface for adding notes. The similar UI will be used to comment on existing notes. When adding comments to an existing note, the top corner map view will change to the existing note view, indicating to the user that they are adding comments to that specific note. The UI #2 is dedicated to retrieving the user's notes. The portfolio UI (i.e., UI #3) displays two note lists. It serves as a mobile family album.

Considering that users are often on the move, the UI #1 maintains simplicity, allowing users to effortlessly create their situated documentaries. The tools initially presented to users are subsequently refined based on their suggestions. Users have expressed their desire for the following tools to be included in the design: (1) text input; (2) audio recording; (3) image attachment; (4) video recording; (5) weather; (6) time tagging; (7) location tagging; (8) emoji tool for expressing emotions; and (9) tool for selecting related family members.

These tools empower users to create and enrich their situated documentaries effec-
tively. The UI #2 features augmented reality (AR) directions to guide users to the specified
location, along with an accompanying note related to that location. On the other hand, the
UI #3 showcases family members who have participated in or contributed to the notes.
By including these features and tools, the design aims to enhance user engagement and
overall satisfaction.

<figure><img src="image_6.png" /><figcaption>**Fig. 2.** UI design samples for adding a new/existing note, retrieving my note, and note lists (i.e., UI #1-#3). Map view for adding a new note, retrieving my note, and note lists (UI #4 and #5)</figcaption></figure>

Users have suggested incorporating family member images alongside the place mark-
ers on the map view. This addition would aid them in deciding whether to invest time in
retrieving their situated documentaries.

When a user walks near a note created by other family members, the map view will display all nearby notes. The user can then either tap on an existing note or create a new note at the same location, as demonstrated in the UI #4. In the UI #5, after the user taps the “Notes Made for Me” button or selects a place marker on the map view, more information about the note will be presented to the user.
## 3.2 User Testing

The “Your Notes, My Love” MARS project is a participatory design endeavor, with many of its functions and interaction points being suggested by users at various stages of the design process. Users have expressed their desire to learn more about their family’s experiences. They seek a solution to help them add location and time details to their regular family album, allowing them to identify who shared the images in the family album and understand the thoughts and emotions of family members at the moment when the event was documented. This design research aims to create products and services that align more closely with users’ needs and expectations by incorporating their knowledge and experiences.

After the UIs were created and a simple paper prototype was produced, this study conducted a five-user test in 2023, featuring four female participants and one male participant. The study posed several questions to gauge user perceptions of the MARS, including: (1) usefulness for users; (2) ease of use for users; (3) overall user experience; and (4) any additional needs from the MARS.

The results indicate that users find the “Your Notes, My Love” MARS design to be personal, valuable, desirable, and useful for preserving their family memories. They appreciate the situated documentaries created by their family members and enjoy the multimedia aspect of these documentaries. The combination of text, images, audio, and videos with emotional content fosters a strong sense of togetherness with their family across time and generations. Multimedia notes help them understand the first-person experiences better. Furthermore, the push notifications bring them pleasant surprises while walking in the city. They can instantly see which family member documented their personal stories at that specific location. The map view provides a strong visual for them to trace their family memories.

One female participant expressed her liking for the MARS design, as it would enable her to learn her grandmother’s cake recipe. She also believes that the MARS design allowed her to learn more about her grandparents’ wisdom. She suggested having a different UI design specifically tailored for elder users. The male participant commented that he would enjoy using the MARS to revisit the city of his birth and discover more about his personal stories. He appreciated the idea that his parents left him a note to share their experience on the day he was born. Another female participant suggested incorporating a family tree feature, which would allow users to gain a better understanding of the relationships among family members.

## 3.3 Discussion

The “Your Notes, My Love” MARS project is mapping experiences for the user and mapping experiences created by the user. A user has proposed incorporating a family tree feature, along with a heatmap on a world map to visualize where the notes are created and interacted with. The notes on the map offer a clear view of the family’s geographical connections, providing a strong visual representation for users to trace their family roots.

Considering the project is an online and onsite family album on the map, one additional consideration is determining where to initiate and conclude this family map. Given
that family is a social construct, the design may need to reconsider the onboarding process to determine who and how individuals can join this mapping project as a family. Additionally, the design should address the question of when and by whom the final relevance notes should be deleted for a family member who has passed away.

# 4 Conclusion and Future for the MARS Design

## 4.1 Conclusion

For this “Your Notes, My Love” MARS, the results show that users seem to find it useful and easy to use to enjoy the “situated documentaries” from the family networks at the location. This MARS design aims to fulfill users’ needs to share love, information, memories, and knowledge within the family. The family context is selected because the design researcher understands that all families have a desire to keep family memories for their family members. Mobile media users like to share their activities with their networks but not with strangers. MARS provides an opportunity for users to document the family moments at the location. It also keeps those family love, information, memories, and knowledge online and onsite for family members to retrieve.

## 4.2 Future for the MARS Design

“Your Notes, My Love” presents a preliminary design concept of MARS technology, aiming to revolutionize the way user-generated mobile situation documentaries are accessed and shared. In today’s digital age, such situated documentaries have become increasingly popular as people capture and share their unique experiences at the location. However, despite their growing popularity, there remains a noticeable lack of innovative designs that facilitate the seamless retrieval of these personal situational documentaries while on the move. The future of this study involves building a functional prototype and testing it with more users. Additionally, more research on users is needed to understand their perceptions, motivations, and needs regarding the “Your Notes, My Love” MARS design. By doing so, this design research project can refine the design and implement features that resonate with users, ensuring that “Your Notes, My Love” becomes a useful, easy-to-use, and valuable tool for accessing and sharing mobile situation documentaries within families.

# References

1. Aarseth, E.J.: Cybertext: Perspectives on Ergodic Literature. The Johns Hopkins University, Baltimore (1997)

2. Angus, A., et al.: Urban social tapestries. IEEE Pervasive Comput. 7(4), 44–51 (2008). https://doi.org/10.1109/MPRV.2008.84

3. Carey, J.W.: Communication as Culture: Essays on Media and Society. Unwin Hyman, UK (1989)

4. Chen, Y.-F.: Design a mobile augmented reality system to facilitate users’ companionship needs. In: Arai, K. (ed.) FTC 2021. LNNS, vol. 358, pp. 12–28. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-89906-6_2
5. Dacko, S.G.: Enabling smart retail settings via mobile augmented reality shopping apps. Technol. Forecast. Soc. Change. **124**, 243–256 (2017). https://doi.org/10.1016/J.TECHFORE. 2016.09.032

6. Fortunati, L.: Italy: stereotypes, true and false. In: Katz, J.E., Aakhus, M.A. (eds.) Perpetual Contact: Mobile Communication, Private Talk, Public Performance, pp. 42–62. Cambridge University Press, Cambridge (2002)

7. Girardin, F., et al.: Digital footprinting: uncovering tourists with user-generated content. IEEE Pervasive Comput. **7**(4), 36–44 (2008). https://doi.org/10.1109/MPRV.2008.71

8. Goodchild, M.F.: Citizens as sensors: the world of volunteered geography. Geoj. **69**(4), 211–221 (2007). https://doi.org/10.1007/S10708-007-9111-Y

9. Haklay, M., Weber, P.: OpenStreet map: user-generated street maps. IEEE Pervasive Comput. **7**(4), 12–18 (2008). https://doi.org/10.1109/MPRV.2008.80

10. Herfort, B., et al.: The evolution of humanitarian mapping within the OpenStreetMap community. Sci. Rep. **11**(1), 1–15 (2021). https://doi.org/10.1038/s41598-021-82404-z

11. Hollerer, T., et al.: Situated documentaries: embedding multimedia presentations in the real world. In: Third International Symposium on Wearable Computers, pp. 79–86. IEEE, Piscataway (1999). https://doi.org/10.1109/ISWC.1999.806664

12. Koskinen, I.K.: User-generated content in mobile multimedia: empirical evidence from user studies. In: Proceedings - IEEE International Conference on Multimedia and Expo, vol. 2, pp. II645–II648 (2003). https://doi.org/10.1109/ICME.2003.1221699

13. Kounavis, C.D., et al.: Enhancing the tourism experience through mobile augmented reality: Challenges and prospects. Int. J. Eng. Bus. Manag. **4**(1), 1–6 (2012). https://doi.org/10.5772/ 51644

14. Lee, K.: Augmented reality in education and training. TechTrends **56**(2), 13–21 (2012). https:// doi.org/10.1007/S11528-012-0559-3

15. Linaza, M.T., et al.: Evaluation of mobile augmented reality applications for tourism desti- nations. Inf. Commun. Technol. Tour. **2012**, 260–271 (2012). https://doi.org/10.1007/978-3- 7091-1142-0_23

16. Ling, R.S.: Taken for Grantedness: The Embedding of Mobile Communication into Society. MIT Press, Cambridge (2012)

17. Ling, R.S.: The Mobile Connection: The Cell Phone's Impact on Society. Morgan Kaufmann, San Francisco (2004)

18. Ling, R.S., Yttri, B.: Hyper-coordination via mobile phones in Norway. In: Katz, J.E., Aakhus, M.A. (eds.) Perpetual Contact: Mobile Communication, Private Talk, Public Performance, pp. 139–169. Cambridge University Press, Cambridge, UK (2002)

19. Mäkelä, A., et al.: Joking, storytelling, artsharing, expressing affection: a field trial of how children and their social network communicate with digital images in leisure time. In: Con- ference on Human Factors in Computing Systems – Proceedings, pp. 548–555 (2000). https:// doi.org/10.1145/332040.332499

20. Matney, L.: Google shows off ARCore, its answer to Apple's ARKit (2017). https://techcr unch.com/2017/08/29/google-shows-off-arcore-its-answer-to-apples-arkit/

21. Nincarean, D., et al.: Mobile augmented reality: the potential for education. Procedia - Soc. Behav. Sci. **103**, 657–664 (2013). https://doi.org/10.1016/J.SBSPRO.2013.10.385

22. Pavlik, J.V.: Experiential media and transforming storytelling: a theoretical analysis. J. Creat. Ind. Cult. Stud. **3**, 46–67 (2018)

23. Pavlik, J.V.: Journalism in the Age of Virtual Reality: How Experiential Media are Transforming News. Columbia University Press, New York (2019)

24. Pavlik, J.V., Bridges, F.: The emergence of augmented reality (AR) as a storytelling medium in journalism. Journal. Commun. Monogr. **15**(1), 4–59 (2013). https://doi.org/10.1177/152 2637912470819
25. Pousneh, A.: Augmented reality in retail: a trade-off between user's control of access to personal information and augmentation quality. J. Retail. Consum. Serv. **41**, 169–176 (2018). https://doi.org/10.1016/J.JRETCONSER.2017.12.010

26. Rauschnabel, P.A., et al.: An adoption framework for mobile augmented reality games: the case of Pokémon Go. Comput. Human Behav. **76**, 276–286 (2017). https://doi.org/10.1016/j.chb.2017.07.030

27. Rikakis, T., et al.: Experiential media and digital culture. Comput. (Long. Beach. Calif.). **46**(1), 46–54 (2013). https://doi.org/10.1109/MC.2012.391

28. Väätäjä, H.K., et al.: Exploring augmented reality for user-generated hyperlocal news content. In: Conference on Human Factors in Computing Systems – Proceedings, vol. 2013-April, pp. 967–972 (2013). https://doi.org/10.1145/2468356.2468529

29. Vera, F., Sánchez, J.A.: A model for in-situ augmented reality content creation based on storytelling and gamification. In: Proceedings of the 6th Mexican Conference on Human-Computer Interaction (2016). https://doi.org/10.1145/2967175

30. Wetzel, R., et al.: Designing mobile augmented reality games. Handb. Augment. Real. 513–539 (2011). https://doi.org/10.1007/978-1-4614-0064-6_25