# Supporting Multiple Perspectives on 3D Museum Artefacts through Interoperable Annotations

Jane Hunter, Chih-Hao Yu

► To cite this version:

Jane Hunter, Chih-Hao Yu. Supporting Multiple Perspectives on 3D Museum Artefacts through Interoperable Annotations. Second IFIP TC 14 Entertainment Computing Symposium (ECS) / Held as Part of World Computer Congress (WCC), Sep 2010, Brisbane, Australia. pp.149-159, 10.1007/978-3-642-15214-6_15 . hal-01056346

HAL Id: hal-01056346

https://inria.hal.science/hal-01056346

Submitted on 18 Aug 2014

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Supporting Multiple Perspectives on 3D Museum Artefacts through Interoperable Annotations

Jane Hunter and Chih-hao Yu

School of ITEE, The University of Queensland,
St Lucia, Queensland, Australia 4072
{j.hunter@uq.edu.au, chih.yu@uqconnect.edu.au}

**Abstract.** Increasing numbers of museums and cultural institutions are using 3D laser scanning techniques to preserve cultural artefacts as 3D digital models, that are then accessible to curators, scholars and the general public via Web interfaces to online galleries. Museums are finding the cost of providing metadata for such collections prohibitive and are keen to explore how they might exploit Web 2.0 social tagging and annotation services to capture community knowledge and enrich the contextual metadata associated with their collections. Although there exist some annotation services for 3D objects, they are designed for specific disciplines, not Web-based or depend on proprietary software and formats. The majority also only support the attachment of annotations to whole objects – not points, 3D surface regions or 3D segments. This paper describes the 3DSA (3D Semantic Annotation) system developed at the University of Queensland that enables users to attach annotations to 3D digital artefacts. The 3DSA system is based on a common interoperable annotation model (the Open Annotations Collaboration (OAC) model) and uses ontology-based tags to support further semantic annotation and reasoning. This common approach enables annotations to be re-used, migrated and shared – across annotation clients and across different 3D and 2.5D digital representations of the one cultural artifact. Such interoperability is essential if cultural institutions are to easily harness knowledge from a broad range of users, including curators, students and remote Indigenous communities, with different client capabilities.

**Keywords:** 3D annotations, tags, semantics, interoperability, ontologies

## 1 Introduction and Objectives

Advances in 3D data acquisition, processing and visualization technologies are providing museums and cultural institutions with new methods for preserving cultural heritage and making it more accessible to scholars, traditional owners and the public, via online search interfaces. Increasing numbers of museums are using 3D scanning techniques to overcome the limitations of 2D data representations and to improve access to high quality surrogates of fragile and valuable artefacts via the Internet [1-4]. The trend is increasingly towards the use of 3D laser scanners to capture precise 3D digital models that can be accurately analysed, measured and compared. However
there are a number of challenges that come with building online collections of 3D museum objects, making them accessible to different types of users and enabling their classification and the attachment of community knowledge through tags and annotations.

Firstly, the file size of the 3D digital objects is often problematic for many users who are unable to quickly and easily download and render the objects due to limited bandwidth, CPU, graphics cards or the need for specific 3D rendering software. Secondly, as the size of online collections of 3D artefacts grows, the ability to enable search and browsing across these distributed repositories becomes more difficult. Museums are finding the cost of providing metadata and rich contextual information for their collections prohibitive and are keen to explore how they might exploit social tagging and annotation services [5]. High quality tags and annotations – attached to both the complete object as well as to specific segments or features – have the potential to significantly improve the relevance of retrieved search results. Although there already exist some annotation services for 3D objects, they are designed for specific disciplines or depend on proprietary software and formats. The majority also only support the attachment of annotations to the whole objects – not to 3D points, surface regions, 3D parts or segments (e.g., the handle on a pot).

Hence the aims of the work described here are to develop services to support the following:

* Workflows for streamlining the generation of multiple alternative digital representations (high resolution, medium resolution and low resolution) of each 3D museum object in high-quality, standardized and widely used formats;

* Web-based, easy-to-use, 3D tagging/annotation tools that support the attachment of annotations to points, surface regions or 3D segments (i.e., meaningful parts or features) on a 3D model. The difficulty lies in specifying the particular feature of interest via simple drawing, selection and segmentation tools. For example, drawing the boundary of a 3D surface feature or a 3D segment can be very difficult and time consuming.

* Tagging and annotation tools that enable annotations/tags to be automatically attached to, migrated between and displayed, for different digital versions (high, medium, low resolutions) of each museum artifact;

* Semantic annotation tools – that use machine-understandable tags drawn from an ontology. Our aim is to use the CIDOC-CRM [7] ontology, which has been designed specifically for the museum community, but to extend it with discipline-specific sub-ontologies (e.g., Indigenous Carvings). Ontology-based annotations are valuable because, in addition to supporting validation and quality control, they allow reasoning about the annotated resources, enabling it to become part of the larger Semantic Web;

* A common model for attaching annotations to 3D artefacts regardless of their format. Such a model also enables re-use and display of annotations across different annotation clients. Our aim is to evaluate the Open Annotations Collaboration (OAC) model for supporting the migration of annotations between multiple versions of a 3D object and using different annotation clients.
## 2 Related Work

Most prior work in the field of 3D annotations has focused on the annotation of discipline-specific objects – for example, architectural and engineering CAD drawings [7,8], 3D crystallography models [26] and 3D scenes [27]. All of these systems enable users to attach annotations to 3D models and to browse annotations added by others, asynchronously. However they are all limited to the discipline-specific format of the target objects. Adobe Acrobat 3D also provides a 3D JavaScript API that allows annotation of 3D CAD models or U3 objects stored in PDF using a proprietary SDK. However the documentation is lengthy and attaching annotations is a programmatic exercise [23]. A survey of existing systems failed to reveal any interoperable, collaborative, Web-based annotation systems for 3D models of museum artefacts, that enable descriptive text or semantic tags to be attached (either to the whole object or a point or region on the object) – and then saved to enable later, asynchronous searching, browsing and response, by other users.

Projects such as SCULPTEUR [9], the Princeton 3D search engine [10] and Columbia Shape Search [15] use a combination of machine learning (to extract colour, pattern and shape) and application semantics (who, what, where, when etc.) to automatically cluster 3D objects. However these projects fail to take advantage of community-generated tags and annotations drawn from ontology-directed folksonomies. Hunter et al [14, 15, 16] have previously applied semantic inferencing rules to enable the automated high-level semantic annotation of 2D images from low-level automatically-extracted features – and demonstrated improvements in concept-based search performance. Hunter et. al. have also developed annotation tools for 3D museum artefacts, based on the Annotea model [13]. But this previous work has only enabled the attachment of tags and comments to 3D points and/or views of the complete object.

Other relevant prior work is in the area of segmentation of 3D models [22, 28] and the attachment of semantic annotations to segments [29, 30]. Previous segmentation approaches have involved mesh segmentation approaches and manual or user-guided approaches [12, 31]. The ShapeAnnotator [22] focuses on automatic segmentation to separate a 3D object into different segments. The ShapeAnnotator is not Web-based and does not enable users to annotate whatever region they select – only pre-identified segments. It also does not display textures for 3D models, which makes unattractive to cultural institutions. Our aim is to adopt an approach similar to Ji et al [12] to perform the automated segmentation and to apply and evaluate it within the context of a specific collection of 3D museum artefacts.

As far as we are aware, there are currently no open-source tools that: enable Web-based semantic annotation of 3D museum objects; that use ontology-based semantic tags; or that enable easy tagging of points, surface regions or 3D parts on a 3D digital object. Finally we are unaware of any system that enables the easy migration of tags/annotations between different digital versions of the one 3D object – a critical requirement if museums are going to engage with users from a range of different communities and with access to variable computer capabilities.
## 2.1 The Open Annotations Collaboration (OAC) Data Model

The lack of robust interoperable tools for annotating across heterogeneous repositories of digital content and difficulties sharing or migrating annotation records between users and clients – are hindering the exploitation of digital resources by many scholars. Hence the Open Annotations Collaboration (OAC) was established to facilitate the emergence of a Web and Resource-centric interoperable annotation environment that allows leveraging annotations across the boundaries of annotation clients, annotation servers, and content collections. To this end, an annotation interoperability specification consisting of an Annotation Data Model (Fig. 1) has been developed. Fundamental principles that were adopted by the OAC include:

* The core entities of an annotation (*Annotation*, *Content*, *Target*) are independent Web resources that are URI-addressable and hence discoverable and re-usable;

* The Annotation *Target* (the object being annotated) and the Annotation *Content* (the body of the annotation) can each be any media type;

* Annotation Targets and Content are frequently segments of Web resources;

* The Content of a single annotation may apply to multiple Targets or multiple annotation Contents may apply to a single Target;

* Annotations can themselves be the Target of further Annotations.

<figure><img src="image_2.png" /><figcaption>Fig. 1. The Alpha OAC Data Model [6]</figcaption></figure>

The case study in which different users from different backgrounds (curators, scholars, students, public) annotate and tag a museum object (represented in different digital formats) – and then share and aggregate those annotations – is an ideal case study for evaluating the OAC model. Fig. 2 below illustrates how the OAC is relevant to the 3DSA application through a simple example in which user “jane” attaches the textual annotation “Wik totem” to a point on a 3D object. This annotation is automatically migrated across the three digital representations of this object.
<figure><img src="image_3.png" /><figcaption>Fig. 2. Instance of the OAC model within the 3DSA application</figcaption></figure>

## 3 Case Study

To evaluate the proposed annotation services, we are currently using Indigenous wooden ceremonial sculptures from the Wik peoples of Western Cape York. This collection of wooden, ochre-painted sculptures is held in the UQ Anthropology Museum. Indigenous artists from Cape York are interested in emulating and extending the techniques used by artists from these earlier periods. They would like to be able to access high resolution 3D versions of the sculptures without having to travel to Brisbane for long periods. In addition, the UQ Art Museum is developing an exhibition around these sculptures, to open in 2010. The aim of this project is to work with the UQ Anthropology Museum curators, Indigenous artists from Cape York and the UQ Art Museum to develop a virtual collection of 3D models which can be used for remote access, collaborative annotation, knowledge sharing, exhibition development and the evaluation of this project's outcomes.

## 4 Implementation

### 4.1 Generating the 3D Models

The first phase of the project involves scanning each artefact using a Konica Minolta Vivid9i non-contact 3D laser scanner and generating the different 3D digital models using GeoMagic software. Each museum artefact is initially scanned into a VRML format, stored in Collada format using Autodesk Maya and converted into O3D format (Google's 3D scene graph API) using Google's converter. The 2.5D file (FlashVR format) is generated by capturing a series of images from the O3D file. At this stage, the project has generated a sample set of Indigenous artefacts for evaluation purposes. More artefacts from a variety of backgrounds and materials, will be scanned in the future to more fully evaluate the search and indexing features.

A high quality 3D digital model for a cultural heritage artefact contains 50-200 MB of data and 20-50,000 polygons. However, many users don't have the bandwidth,
computational power or graphics cards capable of downloading and rendering such objects in a timely fashion – or enabling real-time interaction (panning, rotating, zooming) with the 3D models. In order to support users with limited computation power or bandwidth, we generate four different representations of each artefact:

* Archival quality 3D model (Raw 3D data): for storage purposes only, not accessible online.

* High quality 3D model for users who have standard CPU and internet speed.

* Low quality 3D model - compressed version for users with limited CPU, graphics card or slow internet.

* 2.5D VR object - non-3D version, suitable for users who do not have a graphical processor and with a very slow internet connection.

## 4.2 System Design and Implementation

Using the 3D objects generated via the scanning and conversion process described above, we have developed a Web interface to a gallery of objects. Users can search and browse the gallery of 3D objects via thumbnails, a tag cloud and keyword search. The 3DSA annotation prototype (shown in Fig. 4) is also accessible via a link from the project website's 3D gallery<fnref n="1" />. A high-level view of the system architecture is shown below in Fig. 3.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Screen-shot of the Web-based 3D Annotation prototype 3DSA</figcaption></figure>

The Annotation prototype was developed using a combination of Web 2.0 technologies and third party services including:

* 3D viewer – Google's O3D scene graph API, provides a browser plug-in with a shader-based, low-level graphics API and a high level Javascript library. O3D is flexible, extensible [17], cross-compatible, open source and Google's proposal as an open Web standard for 3D [18].

<footnote n="1">¹ http://www.itee.uq.edu.au/~eresearch/projects/3dsa</footnote>
* 2.5 VR object viewer – this was developed using Adobe Flex, a free, open source framework for building web applications using ActionScript 3.0.

* Annotation storage – is implemented using AJAX and Danno, an HTTP-based repository that provides APIs for creating, updating, deleting and querying annotations and replies, and for bulk upload and harvesting of annotations [19].

* User interface – this was developed using AJAX, PHP and jQuery, a JavaScript Library that simplifies HTML document traversing, event handling, animating, and Ajax interactions for rapid web development [20].

Fig. 4 shows a screen shot of the prototype in use. On the left hand side, are the different versions of the artefact being annotated – users choose the most appropriate for their environment. In the center is the display panel showing the currently selected 3D object and attached annotations. On the right hand side is the annotation search and browse panel. Clicking on the “New Annotation” button displays a new panel that enables users to enter the contextType of the annotation (object, point, region, segment), the Type of annotation (tag, comment, query, feedback) and the body of the annotation. The creator and date/time information is also captured. The complete details/metadata for a chosen annotation are displayed in the bottom centre panel.

<figure><img src="image_5.png" /><figcaption>Fig. 4 Screen shot of the 3DSA Annotation Prototype</figcaption></figure>

After the user enters the annotation data, they can specify the context/point on the object to which it is attached. (Region and Segment annotations are still under development). After saving the annotation, the system then determines if there are other representations of this artifact available. If there are, then the annotation context is re-calculated to automatically migrate the annotation to the other formats (e.g., from 3D high-res to 2.5D Flash). For example transferring a 3D coordinate (X, Y, Z) to 2D coordinate (X, Y), the formula below is implemented [24]:

$$P(X, Y, Z) = LP(X, Y, Z) \times WM \times VM \times PM$$

*P(X,Y,Z) = Point that had been projected on to a 2D screen*

*LP(X,Y,Z) = Local position of the object*

*WM = World matrix, VM = View matrix, PM = Projection matrix*
# 5 Discussion

Feedback from museum curators on the 3DSA system has been extremely positive. They are very excited about the added dimensions and details that 3D scanning can offer but they realize that there are a number of barriers preventing wide-spread adoption of 3D digital formats in online museum collections, including: the time it takes to generate the 3D objects, the size of the files, the time to download and render the files and the graphical and computational power required to support real-time rendering and interactive panning, zooming and rotating. In this paper we firstly describe workflows to streamline the generation of multiple 3D/2.5D digital representations of museum artefacts to satisfy the anticipated range of users and users' computer capabilities, whilst still maximizing quality and minimizing effort. We have then demonstrated a prototype to enable the attachment of community-generated tags and annotations to 3D digital objects. We have also demonstrated how the Open Annotations Collaboration Model can be extended and applied to support interoperability of annotations across clients, across target objects in different digital formats and their attachment to segments (points, regions, 3D parts) of 3D/2.5D digital objects. The complex algorithms for mapping annotations between objects of different resolution have been comprehensively tested and have demonstrated high performance – both speed and accuracy. User evaluations of the annotation prototype have identified a number of improvements to the user interface, including the need for greater consistency in annotation displays between the O3D and Flash plug-ins.

# 6 Future Work and Conclusions

In this paper we describe a Web-based 3D annotation service that does not entail a steep learning curve to create/edit/view annotations attached to 3D digital objects. Annotations can be easily created and attached to 3D digital objects via the O3D or Flash browser plug-ins. Annotations can be attached to 3D objects in different formats (high-res, med-res, low-res (2.5D)) and are automatically migrated between them. This enables both users with access to high-end computers and users with slow computational or graphical processors or limited bandwidth, to use the 3DSA tool and to share annotations across platforms and 3D digital formats.

Future aims include adding support for the newARB format that will be supported by major browsers (Safari, Chrome, Firefox, Opera). We also plan to explore how users attach annotations to surface regions and 3D segments. In addition, we plan to allow tags to be extracted from folksonomies and/or the CIDOC/CRM ontology (ontology-directed folksonomies), to support faceted search. As such, this project differs from projects such as SCULPTEUR, the Princeton 3D search engine and Columbia Shape Search, in which indexing is entirely based on machine learning and semantics but fails to take advantage of folksonomic tags. By combining both user-generated tags and automatic feature extraction, we will significantly enhance the discovery of 3D cultural artefacts and associated content. The outcome will be an online digital repository/gallery of 3D cultural heritage artefacts enriched with both
manually-generated and automatically-generated metadata to enable fast accurate
search and retrieval of 3D objects by both museum experts and the general public.

Finally, we have begun experimenting with the QR codes [33] to enable museum
visitors to retrieve community-generated annotations via their iPhones. QR codes
(small printable tags) can be generated from the 3DSA web pages, and attached to the
physical museum artefacts in the exhibition. This enables museum visitors with the
QR code app on their iPhone, to retrieve the related 3DSA web page that displays the
3D digital version with aggregated annotations. This is a very exciting development
with significant potential that will also require further testing and evaluation.

To conclude, the 3DSA system represents a highly innovative approach to cultural
heritage that combines the best of Web 2.0, Semantic Web and mobile technologies to
maximize the preservation, capture, dissemination and re-use of knowledge about
cultural heritage.

**Acknowledgments.** The work described in this paper has been funded by Microsoft Research Asia through the e-Heritage program and through a University of Queensland Major Equipment and Infrastructure (MEI) grant. The Open Annotations Collaboration component has been funded by the Andrew W. Mellon Foundation.

References

1. J. Hunter, R. Schroeter, B. Koopman, and M. Henderson, "Using the semantic grid to build bridges between museums and indigenous communities," in Proceedings of the GGF11-Semantic Grid Applications Workshop, June 10, 2004, pp. 46-61.

2. "3D digital preservation of cultural heritages," ikeuchi Lab: University of Tokyo. [Online]. Available: http://www.cvl.iis.u-tokyo.ac.jp/gallery_e/

3. J. Rowe and A. Razdan, "A prototype digital library for 3D collections: Tools to capture, model, analyze, and query complex 3D data," in Museums and the Web 2003, D. Bearman and J. Trant, Eds. Toronto: Archives & Museum Informatics, 2003, pp. 147-158.

4. Isler, V., Wilson, B., and Bajcsy, R. 2006. Building a 3D Virtual Museum of Native American Baskets. In *Proceedings of the Third international Symposium on 3D Data Processing, Visualization, and Transmission (3dpvt'06)* (June 14 - 16, 2006). 3DPVT. IEEE Computer Society, Washington, DC, 954-961.

5. Chun, S, Cherry R, Hiwiller D, Trant J. and Wyman, B 2006. "Steve.museum: An Ongoing Experiment in Social Tagging, Folksonomy, and Museums," Museums and the Web 2006.

6. CIDOC Conceptual Reference Model, ICOM http://cidoc.ics.forth.gr/

7. Jung T., Do E. Y. and Gross M. D., 1999, Immersive Redlining and Annotation of 3D Design Models on the Web, *8th International Conference on Computer Aided Architectural Design Futures*. Kluwer.

8. Jung, T., Gross, M. D., and Do, E. Y. 2002. Annotating and sketching on 3D web models. In *Proceedings of the 7th international Conference on intelligent User interfaces* (San Francisco, California, USA, January 13 - 16, 2002). IUI '02. ACM, New York, NY

9. Addis et al., New Ways to Search, Navigate and Use Multimedia Museum Collections over the Web, in J. Trant and D. Bearman (eds.). *Museums and the Web 2005: Proceedings*, Toronto: Archives & Museum Informatics, published March 31, 2005 at

10. "Princeton 3D model search engine," Princeton Shape Retrieval and Analysis Group. [Online]. Available: http://shape.cs.princeton.edu/search.html
11. C. Goldfeder and P.Allen, Autotagging to Improve Text Search for 3D Models, IEEE Joint Conference on Digital Libraries (JCDL), Pittsburgh, July 2008

12. Ji, Z., Liu, L., Chen, Z., and Wang, G. 2006. Easy mesh cutting. *Computer Graphics Forum* 25, 3 (Sept.), 283–292.

13. R.Schroeter, J.Hunter, J.Guerin, I.Khan, M.Henderson, "*A Synchronous Multimedia Annotation System for Secure Collaboratories*", e-Science 2006, Dec 4-6, Amsterdam, Netherlands

14. L.Hollink, S.Little, J.Hunter, "*Evaluating The Application of Semantic Inferencing Rules to Image Annotation*", Third International Conference on Knowledge Capture, KCAP '05, Oct 2-5, 2005, Banff, Canada

15. J.Hunter, S.Little, "*A Framework to enable the Semantic Inferencing and Querying of Multimedia Content*", International Journal of Web Engineering and Technology (IJWET) Special Issue on the Semantic Web, Vol 2, Nos 2/3, 2005;

16. S.Little, J.Hunter, "*Rules-By-Example - A Novel Approach to Semantic Indexing and Querying of Images*", International Semantic Web Conference ISWC 2004, Hiroshima, Nov 2004;

17. M. Papakipos, "Introducing O3D", April 20, 2009 [Online] Available: http://o3d.blogspot.com/2009/04/toward-open-web-standard-for-3d.html

18. Google O3D official website FAQ, [Online] Available: http://code.google.com/p/o3d/wiki/FAQs

19. Danno/ Dannotate Overview, [Online] Available: http://metadata.net/sites/danno/index.html

20. jQuery Official Website, [Online] Available: http://jquery.com/

21. Vector Math for 3D Computer Graphics, Fourth Revision, July 2009 [Online] Available: http://chortle.ccsu.edu/VectorLessons/vectorIndex.html

22. M. Attene, F. Robbiano, M. Spagnuolo & B. Falcidieno, "Semantic Annotation of 3D Surface Meshes based on Feature Characterization", Genova, CNR, Italy

23. Adobe Acrobat 3D Annotation Tutorial, July 27th 2005 [Online] Available: http://www.adobe.com/devnet/acrobat/pdfs/3DAnnotations.pdf

24. R. Koci, "Computer Graphics Unveiled – World, View and Projection Matrix Unveiled" [Online] Available: http://robertokoci.com/world-view-and-projection-matrix-unveiled/

25. P. Laurila, "Geometry Culling un 3D Englines", Sep 10th, 2000 [Online] Available: http://www.gamedev.net/reference/articles/article1212.asp

26. Hunter, J., Henderson, M., and Khan, I. (2007) Collaborative Annotation of 3D Crystallographic Models. *J. Chem. Inf. Model.*, 47, 6, 2007 http://pubs.acs.org/cgi-bin/article.cgi/jcisd8/2007/47/i06/pdf/ci700173y.pdf

27. Kadobayashi, R., et al, "3D Model Annotation from Multiple Viewpoints for Croquet", Proceedings of the Fourth International Conference on Creating, Connecting and Collaborating through Computing (C5'06), 2006

28. Shamir, A. 2008. A survey on mesh segmentation techniques. *Computer Graphics Forum*.

29. De Floriani, L., Papaleo, L., and Carissimi, N. 2008. A Java3D framework for inspecting and segmenting 3D models. In *Proceedings of the 13th international Symposium on 3D Web Technology* (Los Angeles, California, August 09 - 10, 2008). Web3D '08. ACM, New York, NY, 67-74. DOI= http://doi.acm.org/10.1145/1394209.1394225

30. Attene,M., Robbiano,F., Spagnuolo,M., Falcidieno,B. (2007) Part-based Annotation of Virtual 3D Shapes. In *Proceedings of Cyberworlds '07, spec. sess. on NASAGEM workshop, IEEE Computer Society Press*, pp.427-436

31. Funkhouser T. et al 2004. Modeling by example. *ACM Transactions on Graphics* 23,3, 652–663.

32. The Open Annotation Collaboration Alpha Data Model http://www.openannotation.org/documents/OAC-Model_UseCases-alpha.pdf

33. Tales of Things http://www.talesofthings.com/