# Tick that Box: Interactive Paper Documents

Yomna Abdelrahman, Thomas Kubitza, Katrin Wolf, Norman Pohl, Albrecht Schmidt

<figure><img src="image_1.png" /></figure>

## To cite this version:

Yomna Abdelrahman, Thomas Kubitza, Katrin Wolf, Norman Pohl, Albrecht Schmidt. Tick that Box: Interactive Paper Documents. 15th Human-Computer Interaction (INTERACT), Sep 2015, Bamberg, Germany. pp.376-383, 10.1007/978-3-319-22698-9_25. hal-01609382

HAL Id: hal-01609382

https://inria.hal.science/hal-01609382

Submitted on 3 Oct 2017

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
# Tick that Box: Interactive Paper Documents

Yomna Abdelrahman, Thomas Kubitza, Katrin Wolf, Norman Pohl and

Albrecht Schmidt

VIS, Stuttgart University, Stuttgart, Germany

**Abstract.** Many documents are still printed on paper and their interaction is dis-connected from the digital paper representation. Conductive ink allows to print documents with embedded electrical circuits. We developed several layouts to add interactive elements, such as tick boxes or signature fields, to paper documents. In this paper we propose several techniques to enhance classical paper documents by an additional layer of conductive ink. Through this layer the documents maintain their traditional properties and affordances but receive new in-teractive capabilities that transform them into smart documents. Attached tiny computing elements enable interaction with the paper, such as sensing selec-tions on a form, detecting a signature, or traditional office equipment (e.g. pa-perclip). Additionally to the layout suggestions we show different ways how the computing unit can be connected to the document and how multi-page documents can be connected to a single computing unit.

## 1 Introduction

Printed material and paper documents still play an important role in modern work en-vironments, including offices and fieldwork. People have well understood how to handle paper documents and many organizations have processes in place for distrib-uting paper documents, implementing access control, or destroying copies. Further, the pen usage for annotating document is found to be the most direct, flexible and in-tuitive way [1]. At the same time we are more and more used to digital documents that offer a variety of functions. Hence, paperless dealing with information is highly unlikely, in our approach we investigate smart documents that use conductive ink and a computing component to reduce the gap between physical documents and digital functionality. The prototypes of smart documents we present in this paper can detect different physical interactions. Examples of these interactions are filling form ele-ments with different pens, tearing up paper, punching holes, and stapling pages. Addi-tionally, mechanisms for physical bookmarking and for detecting annotations are in-vestigated. Detection of these interactions with physical documents offers means to active and responsive documents, and it allows reflecting these actions in their respec-tive digital representations. The contribution of this paper is a set of mechanisms and layout designs that enable interaction with paper documents based on conductive ink printing. These mechanisms are experimentally evaluated and the layout designs are provided under creative commons license.
<figure><img src="image_3.png" /><figcaption>Figure 1: Form elements sheet demonstrating 4 mechanisms: crossing out words (a), check-boxes (b, c), signature (d), punch-hole (e), bookmarks (f), text marking (g), and staple (h).</figcaption></figure>

# 2 Related Work

The unlikelihood of paperless office makes it the focus of researches to maintain its affordances yet augment interactivity aspects. In this section we discuss prior work in two different areas aiming to introduce interactivity to paper form using (1) camera based systems and (2) new technologies namely conductive ink to enable printed electronics.

## 2.1 Interactive Paper Documents

Relevant research with the aim to add and link digital value to paper documents includes work that focuses on linking the physical and digital world and enhancing the responsiveness of paper and office tools[2]–[4]. Through embedding cameras onto the pen[5] or above the desktop[6], [7] for capturing pen-paper interaction, or even using scanners[8]. The Anoto pen<fnref n="1" /> contains a built-in camera to record what has been written on special paper with a preprinted pattern. Further, this pen is employed by other systems to transfer the physical editing on paper documents to the digital world[9]–[12]. CoScribe utilized it to combine work on digital and printed documents, where they used the digital pen to allow annotating, linking and tagging printed and digital documents as the pen interacts with both the printed documents and pen-sensitive displays[13]. In the PaperProof they extended the iPaper framework [4]. However, using a camera as the pen paper interaction tracking device introduced interactive aspect to the papers, however it restricted the interaction to a rather limited area in front of the camera.

## 2.2 Conductive Ink and Printed Electronics

Recently, different forms of conductive ink (paint, pens, and printers) became commercially available² and electronics can nowadays be homemade. Additionally, the

<footnote n="1">¹ http://www.anoto.com
² Bare paint and Circuitworks conductive pen.</footnote>
vast enhancement introduced by chemical sintering[14] allowed the fast and easy use
of conductive ink for drawing or printing circuits using pens or off-the-shelf inkjet
printers[15], [16]. Hodges et al.[17] added interactive elements through stickers.
Moreover, printable electronics have been combined with other technology as de-
scribed in[18] introducing new applications.

3 Smart Printed Paper Forms

Completing a paper form using a pen is generally perceived as very natural interac-
tion. The mechanisms we propose to turn a paper document into a smart document ex-
ploit as much as possible conventional forms, shapes, methods, and tools known from
common daily routines. Those include shapes and layout known from common docu-
ment forms (e.g., checkboxes), office tools such as staplers, paper-clips, hole-punch-
ers and stamps, as well as natural interaction such as ripping a document apart.

**3.1 Traditional to Digital**

Certain traditional forms in paper documents can easily be transferred to conductive
circuit layouts. Two examples are checkboxes and signature fields. Figure 1b-c shows
two checkbox layouts that are optimized for the use with pencil or pen. Their ground
electrode is arranged on the outer side which makes it easy to create multiple stacked
checkbox items. The signature field (Figure 1d) demonstrates a layout which detects
whether text was written inside. Most of the layouts introduced rely on the same elec-
trical principle: two electrodes arranged in close distance that are connected when a
pencil or conductive pen crosses both, significantly reducing the resistance between
them which is then measured by the computing unit. The ground electrode is however
shared between all conductive elements. This principle can be also applied to detect
crossed out words (Figure 1a). A pencil is often used instead of a pen if actions (e.g.,
crossed checkboxes) should be reversible using a rubber. This works as well for re-
moving pencil lines on conductive elements that again increases the resistance be-
tween two electrodes. Further, a set of conventional office tools can be involved in the
interaction with smart paper documents. We found a stapler to be a surprisingly relia-
ble tool for electrically connecting circuits on multiple pages. Figure 1h depicts a lay-
out which targets the interconnection of circuits on multiple pages. This approach al-
lows placing the computing unit only on one page on which all circuits run together.
Further tools are paper-clips that can be used as temporary selectors, a hole-puncher
for setting the archival state of documents. Table 1 provides an overview of tradi-
tional mechanisms that can be also used to realize digital functionality.

3.2 Attaching a Computing Unit

We propose multiple mechanisms for electrically connecting a computing unit to a
paper document. The connection can be either of permanent or temporal type. A per-
manent connection can be achieved by following the functionality of a sticker. Practi-
cally, this is realized by attaching a computing unit to a document for its whole
lifespan for instance by using conductive z-tape [19]. Temporal connections can be realized in different ways:

- Clip: Attaching a clip that uses mechanical pressure for connecting a computing unit (similar to [20]). This can be a tiny paper clip as well as a clip-board with a mechanical spring holder on top.

- Hole-Puncher: Using a hole-puncher with embedded computing for reading out document information. The document information is read out in the moment of punching the paper.

- Stapler: A computing unit on flexible PCB film (such as Seeduino Film) that provides connectors that are stapled onto matching connectors on a paper document.

## 4 Evaluation

As a proof of concept and to evaluate our mechanisms, we conducted a user study with 12 participants (4 female, 8 male). Each participant filled in a paper form with both, a pencil and a ball-pen including: 3 checkboxes and 3 radio buttons with different gaps between the printed circuit wires sizes (0.2mm / 0.5mm / 1.0mm), cross out 2 words written in different font sizes (Myriad Pro, bold, 16pt / 24pt) and sign with her/his name on a dedicated field. The order of the writing tools was counterbalanced. We additionally asked the participants to staple 10 pages to create a conductive connection among those pages. For evaluating the mechanisms of our layout library, we measured the resistance of the connection created through the pencil/pen. While pencils have been used to draw circuits on common paper they do not work well on photo paper: Only in 15 of 108 cases the pencil created a connection with a resistance smaller than 60MΩ (M=38.9MΩ, SD=14.5). The multi-page connections created by the stapler were conductive in all 108 cases. All mean resistances using a ball-pen are in a range that can be sensed by a microcontroller. By adding an appropriate resistance (ca. 200kΩ), a voltage divider can be built for each input mechanism. In this way the checked/signed state can be sensed using digital inputs of common MCUs. The forms used in the study was stored in a normal office environment and the measurements were retaken in 6 months and they were consistent with the previously measured ones, which implies the paper's maintainability.

## 5 Prototypes

As proof of concept we have created two fully functional prototypes. Each of them implements multiple mechanisms that were introduced in the last sections. Prototype 1 focuses on form elements that can detect pen-based user input on the fly. Prototype 2 demonstrates the detection of text markings, bookmarks and the interconnection of multiple pages.
<table><caption>**Table 1.** Traditional mechanisms and their mapping to digital functionality</caption><thead><tr><td>Mechanism</td><td>Traditional Function</td><td>Digital Function</td></tr></thead><tbody><tr><td>Check box</td><td>Selecting (multiple) items</td><td>Digitally Synchronizing the selection</td></tr><tr><td>Radio box</td><td>Selecting an item from a list of mutually exclusive items</td><td>Synchronizing a single item selection</td></tr><tr><td>Signature</td><td>Assuring the correctness of content provided</td><td>Check if the signature is missing</td></tr><tr><td>Ripping the document</td><td>Destroying the document, making it unreadable</td><td>Deleting the digital document / marking as deleted</td></tr><tr><td>Paper-clip</td><td>Temporarily connecting multiple pages</td><td>Temporarily defining a digital state, e.g. a bookmark</td></tr><tr><td>Stapler</td><td>Permanently connecting multiple pages</td><td>Connecting (circuits on) multiple pages</td></tr><tr><td>Punched holes</td><td>Enabling to archive paper documents</td><td>Archiving the digital document</td></tr><tr><td>Rubber</td><td>Erasing pencil strokes</td><td>Remove electric connection created by a pencil</td></tr><tr><td>Crossing out words</td><td>Excluding inapplicable items/words</td><td>Selecting the non-crossed items</td></tr><tr><td>Text marking</td><td>Adding markers to text</td><td>Adding a marker to the corresponding digital document</td></tr></tbody></table>

## 5.1 Computing Unit

To measure changes in the resistance of conductive elements and to transfer this information to a host system, a computing unit and power source are required. We used the Blidgets platform that was also developed by the authors in a recent parallel project. A Blidget has the dimensions of 25x25x3mm, a weight of 3g which makes typical handling of a paper document remains mostly unaffected. It allows instant wireless communication with computers and smartphones via Bluetooth Low Energy (LE) and it is remotely configurable. It's tiny and at footprint allows it to be easily connected to a circuit on a sheet of paper using for instance z-tape. We use a small 110mAh LiPo battery to power the device, but any power source between 1.8-4V is sufficient. Software on a desktop computer equipped with a USB Bluetooth LE dongle allows to instantly monitor the state of all input ports of a Blidget. Those state changes are then mapped to specific actions related to the digital representation of the paper document. The mapping of a Blidget to a digital representation is configured once through a graphical user interface on the host computer. Afterwards modifications on the paper such as checking a box, signing a field are instantly visualized in the same user interface.
## 5.2 Form Elements

Our single-page prototype primarily focuses on demonstrating the feasibility of conductive form elements that allow detecting user input through a pen. As depicted in Figure 1 it is equipped with the Blidget computing unit and fully functional. Figure 1b and 1c shows two circuit layouts that mimic the visual layout of checkboxes. They are optimized for electrically sensing pen strokes. One computing unit port is needed for each checkbox item as well as the common ground. The distance between the checkbox electrodes in the prototype is 0.2mm. Another form element is to cross out items that are not applicable, figure 1a two electrodes are alternately connected to the letters of each item. Also, one computing-unit port is required for each element that can be crossed out in addition to the common ground. The signature field (Figure 1d) is a specific example for a circuit layout that is intended to detect the existence of hand-written text in input fields. It requires one input port of the computing unit. Finally, a mechanism that is not related to user input with a pen is shown in Figure 1e. A simple circuit on the left of the page is intended to detect punch holes. The software on the host computer uses this information to add an “archived” tag to the digital document.

## 5.3 Text Markings, Bookmarks and Multiple Pages

Research shows that, particularly for reading, paper has inherent advantages over digital documents. To state only some of them, annotating paper documents with a pen is intuitive very flexible, and smoothly integrated with reading. Moreover, paper provides for two-handed interaction and navigation and for creating flexible spatial arrangements. Our second functional prototype demonstrates mechanisms for detecting text markings, setting bookmarks and interconnecting circuits on multiple pages. For detecting text markings we introduce a simple layout that consists of a vertical ground electrode and multiple vertical section electrodes. Here the idea is that a user draws a vertical line to the left and connects the electrodes at the height of the text marking. The resolution depends on the number of used section-electrodes: in our prototype 4 electrodes are used resulting in 4 sections with text markings that can be distinguished. A more advanced circuit layout would be placed within the text area (e.g. multiple rows of electrodes) to detect markings at their original spot. For detecting bookmarks we use a simple paper-clip and an area in the upper right corner. This area consists of two generous electrodes that are electrically connected when a paper clip is attached. In this particular case the traditional functionality of the paper-clip (clipping multiple pages) is ignored and instead it is used as a temporal selector. Conductive pads on the left side and staples are used to electrically connect multiple pages. As depicted in Figure 1h 11 electrodes allow to detect 1 bookmark and 4 text marking sections per page (2 text pages and 1 page for the computing unit). One electrode is the common ground. With the rising number of pages also rises the amount of connectors needed on the left side. Here one approach is to share the same connectors between pages and introducing a mechanism that only connects the ground electrode of the active page to the computing unit. A mechanism for encoding the page number of the active page is then required as well (e.g. binary encoding).
## 5.4 Equipment

For the creation of all printed circuit layouts we have used a Brother DCP-J125 printer, Mitsubishi conductiveink (NBSIJ-MU01) and Mitsubishi paper (NBWF-3GF100) optimized for high conductivity. As ball-pen we used the Mitsubishi Uniball GEL IMPACT UM-153S and for permanently connecting the computing unit to paper we used 3M electrically vertically conductive adhesive tape 9703.

# 6 Conclusion and Future Work

We presented a set of mechanisms that allow adding interactivity to paper documents and maintaining their traditional properties and affordances. Our layout library allows an easy reproduction of all used circuit layouts. An evaluation provides insights into the electrical behavior of differently designed input form layouts and the reliability of stapled multi-page connections. Two functional prototypes prove the concept of multiple proposed mechanisms and demonstrate a feasible approach for attaching a computing unit. We envision more advanced layouts and mechanisms to be created. Those could be collected in a software tool that automatically adds the desired interactive elements into documents and generates multi-page layouts as well as connectors for different computing units.

[1] W. J. Hansen and C. Haas, "Reading and writing with computers: a framework for explaining differences in performance," *Commun. ACM*, vol. 31, no. 9, pp. 1080-1089, 1988.

[2] P. Luff, C. Heath, M. Norrie, B. Signer, and P. Herdman, "Only touching the surface: creating affinities between digital content and paper," in *Proc. CSCW*, 2004, pp. 523-532.

[3] P. Luff, K. Pitsch, C. Heath, P. Herdman, and J. Wood, "Swiping paper: the second hand, mundane artifacts, gesture and collaboration," *Pers. Ubiquitous Comput.*, vol. 14, no. 3, pp. 287-299, 2010.

[4] B. Signer, "Fundamental concepts for interactive paper and cross-media information spaces," ETH, ZURICH, 2005.

[5] T. Arai, D. Aust, and S. E. Hudson, "PaperLink: a technique for hyperlinking from real paper to electronic content," in *Proc. CHI*, 1997, pp. 327-334.

[6] P. Wellner, "The DigitalDesk calculator: tangible manipulation on a desk top display," in *Proc. UIST*, 1991, pp. 27-33.

[7] W. Newman and P. Wellner, "A desk supporting computer-based interaction with paper documents," in *Proc. CHI*, 1992, pp. 587-592.
[8] J. M. Heiner, S. E. Hudson, and K. Tanaka, "Linking and messaging from real paper in the Paper PDA," in *Proc. UIST*, 1999, pp. 179–186.

[9] T. Tsandilas, C. Letondal, and W. E. Mackay, "Mus ink: composing music through augmented drawing," in *Proc. CHI*, 2009, pp. 819–828.

[10] C. Liao, F. Guimbretière, K. Hinckley, and J. Hollan, "Papiercraft: A gesture-based command system for interactive paper," *ACM TOCHI*, p. 18, 2008.

[11] F. Guimbretière, "Paper Augmented Digital Documents," in *Proc. UIST*, 2003, pp. 51–60.

[12] C. Winkler, J. Seifert, C. Reinartz, P. Krahmer, and E. Rukzio, "Penbook: Bringing pen+ paper interaction to a tablet device to facilitate paper-based workflows in the hospital domain," in *Proc. ITS*, 2013, pp. 283–286.

[13] J. Steimle, O. Brdiczka, and M. Muhlhauser, "CoScribe: integrating paper and digital documents for collaborative knowledge work," *Learn. Technol. IEEE Trans.*, vol. 2, no. 3, pp. 174–188, 2009.

[14] T. Yoshiki, S. Shino, and K. Kobayashi, "Process for preparing conductive material." Google Patents, 2011.

[15] S. Olberding, N.-W. Gong, J. Tiab, J. A. Paradiso, and J. Steimle, "A cuttable multi-touch sensor," in *Proc. UIST*, 2013, pp. 245–254.

[16] M. E. Karagozler, I. Poupyrev, G. K. Fedder, and Y. Suzuki, "Paper generators: harvesting energy from touching, rubbing and sliding," in *Proc. UIST*, 2013, pp. 23–30.

[17] S. Hodges, N. Villar, N. Chen, T. Chugh, J. Qi, D. Nowacka, and Y. Kawahara, "Circuit stickers: peel-and-stick construction of interactive electronic prototypes," in *Proc. CHI*, 2014, pp. 1743–1746.

[18] J. Sarik, A. Butler, J. Scott, S. Hodges, and N. Villar, "Combining 3D printing and printable electronics."

[19] Y. Kawahara, S. Hodges, B. S. Cook, C. Zhang, and G. D. Abowd, "Instant inkjet circuits: lab-based inkjet printing to support rapid prototyping of UbiComp devices," in *Proc. Ubicomp*, 2013, pp. 363–372.

[20] C. Decker, M. Beigl, A. Eames, and U. Kubach, "DigiClip: Activating physical documents," in *Proc. Distr. Comp. Sys. WS*, 2004, pp. 388–393.