Augmented reality cross-domain solution for physically disconnected security domains

A method comprising the steps of: displaying primary data having a first sensitivity level on a first display screen that is operatively coupled to a first computer; capturing an image of the first display screen with an image capture device that is operatively coupled to a second computer that is communicatively isolated from the first computer such that no data is shared between the first and second computers; executing with the second computer a display recognition and characterization algorithm to recognize the primary data based only on the captured image of the first display screen; and augmenting the primary data by displaying secondary data on a second display, wherein the secondary data is related to, and has a higher sensitivity level than, the primary data.

BACKGROUND OF THE INVENTION

Augmented reality and virtual display technologies are rapidly advancing due to widespread accessibility of handheld devices with built-in cameras, inexpensive gyroscope, accelerometer, and camera sensors, and the ready availability of sophisticated image processing algorithms. The advent of application marketplaces has demonstrated a plethora of text and image processing apps which use sophisticated algorithms to recognize and augment real-world environments. There are also many successful implementations of text and image processing applications within the prior art. Many of the challenges of such systems relate to the difficultly of processing and recognizing objects in the physical environment.

SUMMARY

Described herein is a method for augmenting the display of information having a first sensitivity level with information having a higher sensitivity level comprising the following steps. The first step provides for displaying primary data having a first sensitivity level on a first display screen that is operatively coupled to a first computer. The next step provides for capturing an image of the first display screen with an image capture device that is operatively coupled to a second computer that is communicatively isolated from the first computer such that no data is shared between the first and second computers. The next step provides for executing with the second computer a display recognition and characterization algorithm to recognize the primary data based only on the captured image of the first display screen. The next step provides for augmenting the primary data by displaying secondary data on a second display, wherein the secondary data is related to, and has a higher sensitivity level than, the primary data.

The method for augmenting the display of information having a first sensitivity level with information having a higher sensitivity level may also be described as comprising the following steps. The first step provides for displaying primary data having a first sensitivity level on a first display screen that is operatively coupled to a first computer. The next step provides for capturing live video data of the first display screen with a camera that is operatively coupled to a second computer that is communicatively isolated from the first computer such that no data is shared between the first and second computers. The next step provides for executing with the second computer a display recognition and characterization algorithm to recognize the primary data based only on the video data. The next step provides for augmenting the primary data by displaying secondary data on a wearable, heads-up display viewable only to a user, wherein the primary and secondary data are both within the user's field of view when the user views the first display screen through the head-up display, and wherein the secondary data is related to, and has a higher sensitivity level than, the primary data.

Also described herein is a system for augmenting the display of information having a first sensitivity level with information having a higher sensitivity level comprising a first computer, a first display, an image capture device, a second computer, and a second display. The first computer is configured to process only data having up to the first sensitivity level. The first display screen is operatively coupled to the first computer and configured to display primary data having a sensitivity up to the first sensitivity level. The image capture device is configured to capture an image of the primary data displayed on the first display screen without connecting to, authorization by, or knowledge of the first computer. The second computer is operatively coupled to the image capture device and is configured to process data having up to a second sensitivity level. The second sensitivity level is higher than the first sensitivity level. The second computer is communicatively isolated from the first computer such that no data is shared between the first and second computers. The second display is operatively coupled to the second computer. The second computer is configured to execute a display recognition and characterization algorithm to recognize the primary data based only on the captured image of the primary data on the first display screen. The second computer is further configured to augment the primary data by displaying secondary data having a sensitivity up to the second sensitivity level on the second display.

DETAILED DESCRIPTION OF EMBODIMENTS

The methods and systems described herein are for augmenting the display of information having a first sensitivity level with information having a higher sensitivity level. The system and method relate to augmented reality (AR) technology that provides an “overlay” of additional content which appears adjacent to and/or “on top of” display screens which are visible in a user's field of vision. The disclosed methods and system below may be described generally, as well as in terms of specific examples and/or specific embodiments. For instances where references are made to detailed examples and/or embodiments, it should be appreciated that any of the underlying principles described are not to be limited to a single embodiment, but may be expanded for use with any of the other methods and systems described herein as will be understood by one of ordinary skill in the art unless otherwise stated specifically.

Within some computing environments information is required to be physically separated due to the sensitivity of the information. Information might be deemed sensitive due to impact that disclosure of the information may have; such as harmful impacts on the environment, facilities, operations, or personnel. This is particularly relevant for the protection of proprietary business information in a competitive business environment, protection of tactical military operational information in hostile wartime environment, protecting personal medical information from inadvertent public disclosure, and many other instances where guarantees of privacy are required to protect people, facilities, operations, or information systems from harm or destruction. In general, physical separation between computing systems (an air-gap) is one of the strongest way of guaranteeing that information will not leak or be inadvertently disclosed.

FIG. 1is an illustration of an embodiment of a system10comprising a first computer12, a first display14, an image capture device16, a second computer18, and a second display20. The first computer12is configured to process only data having up to the first sensitivity level. The first display screen14is operatively coupled to the first computer12and is configured to display primary data22of only the first sensitivity level. The image capture device16is configured to capture an image of the primary data22displayed on the first display screen14without connecting to, authorization by, or knowledge of the first computer12. The second computer18is operatively coupled to the image capture device16and is configured to process data having up to a second sensitivity level. The second sensitivity level is higher than the first sensitivity level. The second computer18is communicatively isolated from the first computer12such that no data is shared between the first and second computers12and18respectively. The second display20is operatively coupled to the second computer18. The second computer18is configured to execute a display recognition and characterization algorithm to recognize the primary data based only on the captured image of the primary data on the first display screen14. The second computer18is further configured to augment the primary data by displaying secondary data24of the second sensitivity level on the second display20.

FIG. 2is a flowchart of an embodiment of a method26for augmenting the display of information having a first sensitivity level with information having a higher sensitivity level. Method26comprises, consists of, or consists essentially of the following steps. Step26aof method26provides for displaying primary data having a first sensitivity level on a first display screen that is operatively coupled to a first computer. Step26bof method26provides for capturing an image of the first display screen with an image capture device that is operatively coupled to a second computer that is communicatively isolated from the first computer such that no data is shared between the first and second computers. Step26cof method26provides for executing with the second computer a display recognition and characterization algorithm to recognize the primary data based only on the captured image of the first display screen. Step26dof method26provides for augmenting the primary data by displaying secondary data on a second display, wherein the secondary data is related to, and has a higher sensitivity level than, the primary data.

The first computer12may be any processor capable of processing data and outputting data to the first display screen14. Example embodiments of the first computer12include, but are not limited to: a desktop computer, a laptop computer, a server, a smart phone, and a tablet computer. The first display screen14may be any visual display capable of displaying data to a user. The image capture device16may be any device capable of capturing an image of the first display14and of sharing the captured image with the second computer18without connecting to and/or sharing data with the first computer12. Suitable examples of the image capture device16include, but are not limited to, a digital, still camera and a digital video camera. The second computer18may be any processor capable of receiving the captured image, processing data, and outputting data to the second display20. The second display20may be any display capable of displaying the secondary data24to a user. Suitable examples of the second display20include, but are not limited to: a heads-up display; Google Glass®; a wearable AR device; the Occulus Rift® virtual reality headset paired with suitable software and having an attached camera; and a “smartphone” such as a Google Nexus®, Samsung Galaxy®, or Apple iPhone® having a suitable built-in camera and AR software. The second display20may be hidden from the view of other users such that only the user in possession of the second display20is able to see or otherwise perceive information on the second display20.

The sensitivity levels may correspond with any two sensitivity domains. For example, the primary data22displayed on the first display14may be information that is suitable for viewing by the general public and the secondary data24displayed on the second display20may be proprietary information. In another example, the primary data22may correspond to unclassified data and the secondary data24may correspond to SECRET data, according to a government information security classification system. In another example, the primary data22may correspond to SECRET data and the secondary data24may correspond to TOP SECRET data, as those terms are used in a government information security classification system. In another example, the primary data22may be information that is suitable for viewing by the general public and the secondary data24may include personally identifiable information. In another example, the primary data22may be information that is suitable for viewing by the general public and the secondary data24may be client-health information.

The systems and methods described herein may be used in a number of environments such as multi-classification or “multi-domain” environments with both classified and unclassified material which must be processed and displayed by physically separate computer systems. Other suitable environments include, but are not limited to: coalition environments with many individuals with differing access levels in addition to shared and common computer systems and displayed content; North Atlantic Treaty Organization (NATO) environments with similar access restrictions and shared information requirements; law enforcement digital forensics environments where law enforcement systems cannot physically connect to systems which are being examined due to the potential for contaminating information on such systems; and medical environments where patient data cannot be disclosed due to restrictions in the Health Insurance Portability and Accountability Act (HIPAA) but where the patient data should be available and displayed in context when a doctor performs research on patient health issues.

Other uses of the systems and methods described in the claims include, but are not limited to, the provision of a personal AR system that provides personalized views which augment shared/common displays of: patient health data in a common area without disclosing patient identities (where an individual patient identity associated with a given set of health data is displayed on the second display20); computer displayed commercial advertisements (to highlight items of potential individual interest without providing companies with information of such interest); display of flight schedules (e.g. to highlight flights, cities, or the flights of known associates or family members). In each of these use cases data is neither shared nor transmitted between the second computer18and the first computer12.

The systems and methods claimed herein enable a novel application of AR display capabilities in which the second computer18is used to augment the first display14and where the two computer devices (the first and second computers12and18respectively) are not connected and cannot share or transmit data. Because of the lack of “backchannel” data sharing between the devices we have taken a different approach to object recognition and augmented display, relying on both text and image processing as well as optional dynamic visual codes which are displayed on the first display14and recognized by specialized algorithms running on the second computer18. The claimed methods and systems enhance a user's view of the first display14which otherwise would not be possible because the first display14and the second display18are physically separate and do not communicate via a computer network or direct connection (e.g. physical cable, WiFi, Bluetooth, etc.).

FIG. 3is an illustration of system10wherein the first display14is further configured to display an optional visual marker28. The visual marker28may be any visual code that can be recognized by the second computer18from the image of the first display14captured by the image capture device16. As used herein, suitable embodiments of the visual marker28may include, but are not limited to, any of the following: computer vision (CV) markers; matrix codes; two-dimensional bar codes; CyberCode™; visual tags; visual codes; quick reaction (QR) codes; Data Matrix codes; fiducial markers; etc. In a conventional AR approach, visual codes are generally already known by the AR system (e.g. fiducial makers). The second computer18may be configured to extract small amounts of metadata from each marker which may be used as search criteria or for direct use as Universal Resource Indicators (URIs) without connecting to, authorization by, or knowledge of the first computer12. The visual marker28may time-coded in that is may change over time depending on the content of the primary data22. The use of dynamic, on-screen codes allows for the second computer18to display supplementary data on the second display20which cannot be displayed on the first display14due to issues of classification, security, privacy, or an innate difficulty in data transmission between devices.

In one embodiment, system10allows the second computer18to operate in a disconnected manner such that it could be used within a secure facility to augment the display of unclassified primary data22displayed on the first display14with classified secondary data24. In the embodiment of the system10where the second display is a wearable AR display, a first user wearing the AR display may be given access to information (i.e., the secondary data24) which is not displayed on the first display14. A second user (not wearing the AR display) who sees the first display14would not be provided and would not know about the secondary data24displayed only to the first user on the second display20. In another embodiment of the system10, several different users may each have a personal, wearable second display20and the secondary data24displayed on each wearable second display20may be tailored to the respective security clearance/need-to-know of each individual user. In other words, different individuals with separate AR systems could be given different AR displayed information or no AR displayed information in association with the primary data22visible by all the users on the first display14.

InFIG. 3, the second computer18, the second display20, and the image capture device16are all integrated into a lightweight, wearable heads-up-display device30with the ability to process incoming video data from an attached front-facing camera embodiment of the image capture device16. The heads-up-display device30is capable of displaying high-resolution text and images which appear to be embedded within and adjacent to objects within the user's physical environment. Suitable examples of the heads-up-display30are commercially available and would provide the basic image capture and image processing capability when coupled to a sufficiently capable graphics processing unit (GPU) or general-purpose central processing unit (CPU). As such, an embodiment of the heads-up-display30could be implemented using any off-the-shelf AR hardware that provides a sufficiently private viewing experience and sufficient computing capabilities for real-time video processing.

One way of making the heads-up-display30lightweight is to offload video processing to a connected to a general-purpose CPU as might be provided by a handheld computing device (such as a smartphone or similar device). Such a device could also act as a stand-alone storage device to contain all of the data needed to augment surrounding first displays14with classified or personalized information. An embodiment of the heads-up-display30would not rely on visual markers28embedded in the primary data22such as fiducial markers or CV markers. Rather, the aforementioned embodiment of the heads-up-display30would be capable of recognizing objects, text, graphics, and images on any computer display. Reference character32, shown inFIG. 3, corresponds to a portion of text displayed on the first display14that is recognized by the second computer18. Reference character34, shown inFIG. 3, corresponds to a partially-obscured application window displayed on the first display14.

FIG. 4is a flowchart depicting steps of a display recognition and characterization algorithm that may be employed by the heads-up-display30. The first step36provides for identifying display devices based on live video data37that includes the captured image of the first display14. The second step38provides for applying a perspective transform to each identified display to compensate for the variety of angles at which the first display14may be positioned with respect to the image capture device16. The third step40provides for improving image and camera characteristics for improved image quality for each region within the video stream which contains a display device. The fourth step42provides for applying object recognition algorithms to screen content on each video stream region. The fifth step44provides for recognizing on-screen objects which are suitable candidates for augmentation. The sixth step46provides for inserting augmentations into the second display18shown to the user of the heads-up-display30.

More specifically, in reference to the display recognition and characterization algorithm and the flowchart shown inFIG. 4, a general-purpose embodiment of the heads-up-display30would first process the incoming image data to identify first displays14visible within the field of view of the image capture device16. Potential first displays14can be identified using image processing algorithms to identify rectangular and off-angle 4-sided polygon shapes. Peculiarities of common display technologies can also be used to identify regions of the captured image and/or live video data37that are likely to contain images of first displays14. For example, each display technology transmits light with a color gamut which is often specific to the display and the tuning of the display. Known profiles of common display color gamuts can be used to match regions of the captured image and/or a still frame of live video data37as first displays14. Off-angle views of first displays14may be identified using readily available algorithms, such as line detection algorithms, parallel line recognition algorithms, perspective rectangle detection, skew transforms, etc.

Once a potential first display14is identified, on-screen content may not be visible due to focus, exposure, color, and contrast settings of the image capture device16. Each screen rectangle (or polygon) might need specific settings adjustments due to peculiarities in the screen characteristics or due to off-angle viewing characteristics (such as color issues common to liquid-crystal based display devices). For each display screen polygon the image capture device16's exposure and other settings may then be dynamically adapted based on the brightness of an identified first display14or group of first displays14. Automated exposure, white-balance, contrast and other adjustment algorithms are readily available to one having ordinary skilled in the art and knowledgeable of available image processing algorithms. Examples of such algorithms include: contrast factor estimation, intensity normalization, histogram matching, histogram equalization, detail-preserving S-curve adjustment, over-exposure likelihood estimation and compensation, color gamut correction, and others.

In system10, the potential first displays14are identified first so that image quality can be optimized based on the first display14characteristics instead of average environment and lighting characteristics of the entire captured image and/or the live video data37. By first identifying potential display devices, the afore-mentioned algorithms can work optimally based on the brightness, color gamut, and contrast characteristics of each individual display14. Additionally, regions of the captured image that do not contain an image of the display14may be used to estimate ambient lighting conditions in order to inform the image correction algorithms. An embodiment of system10can also incorporate one or more hardware light meters for estimation of ambient lighting conditions. System10may also be configured such that image regions surrounding or otherwise adjacent to each of several individual regions of the captured image can individually inform image correction algorithms of ambient lighting conditions. An embodiment of system10would be capable of independently optimizing several identified screen areas simultaneously and independently so that later steps in image processing can proceed with the best possible image quality for each identified screen. In other words, system10may be configured to segment the captured image into multiple segments if multiple first displays14are identified in the captured image such that each segment corresponds to a respective first display14. If several first displays14are identified, the second computer18may be configured to simultaneously or sequentially apply the object recognition and characterization algorithm to each segment of the captured image. Each segment of the captured image can be used independently to infer display characteristics (such as color gamut) of each individual screen region detected.

Once the first display14is identified and the image quality optimized, the object recognition and characterization algorithm may process the first display14to search for recognizable objects, text, and images (such as software user interface elements, buttons, icons, menus, text, and images). The performance of the object recognition and characterization algorithm may be improved by incorporating knowledge of the software environment of the first computer12. A Microsoft Windows® graphical-user-interface (GUI) uses different display elements than a Mac OS® GUI. Prior knowledge of the GUI environment of the first display14allows the object-recognition and characterization algorithm to correctly and quickly identify on-screen elements while reducing computational overhead. In one embodiment of the object-recognition and characterization algorithm, the second computer18is preloaded with knowledge of the existing fonts on a variety of computer systems so that optical character recognition (OCR) can be accomplished with minimal effort. In other words, the display recognition and characterization algorithm uses an OCR algorithm adjusted to recognize computer-generated fonts. One aspect of the on-display object recognition approach is that knowledge of the viewing angle, skew transforms, and screen optical characteristics (which were learned through the application of image transforms and image processing algorithms) allows the on-screen object recognition and characterization algorithm to adjust its knowledge of expected on-screen objects by applying a perspective transformation to each object's shape and image correction algorithms to each object's optical characteristics (such as color gamut correction). That is, the screen's physical dimensions (as it is seen at a particular angle) imply that each on-screen object will be transformed in the same manner. Similarly, the screen's optical characteristics imply that each on-screen object will be corrected in the same manner. Using this knowledge and knowledge of the GUI environment elements and semantics allows the recognition of on-screen objects to be performed very quickly using minimal computing overhead.

Once on-screen objects are identified, a search may be performed to identify suitable candidates for augmentation. The best candidate objects will contain sufficient information to determine useful data to overlay that is not present on the first computer12. The recognition engine may be identifying text via OCR, images using image and object recognition algorithms, or embedded metadata using CV and fiducial markers. Recognition of computer user interface semantics also allows the heads-up-display30to choose a location for placement of virtual objects that compose the secondary data24. The placement of secondary data24may depend on how a user is interacting with the first computer12and the first display14. For a user sitting in front of or otherwise actively using the first computer12and the first display14, an appropriate location for a virtual object would be chosen so as not to obscure currently displayed primary information22on the first display14or user interface elements of the first display14. For a user standing at some distance from the first display14, obscuring some elements of the user interface or display may be acceptable or even desirable. In some embodiments, obscuring elements of the first display14may be desirable so that corrected or replacement information may be shown to the user instead of the original content.

In the case of open-ended object recognition, many possible augmentations may be possible and selecting a suitable augmentation is often a difficult design choice based on the desired operation and purpose of an embodiment of the system10. For example an embodiment of system10could be designed to augment the first display14by calling up, and presenting on the second display20, classified technical reports that relate directly to the primary data22displayed on the first display14. Other embodiments may display on the second display20patient data (such as a patient history) for patients identified using unique visual markers28embedded within anonymized documents on the first display14. The first computer12may be configured to automatically annotate and tag the primary data22using the visual markers28. Alternatively such system10may be designed to allow a user to manually annotate screen content with appropriate visual markers28. The optional visual markers28allow for more specific augmentations of the first display14as well as more accurate tracking of the screen surface and screen optical characteristics of the first display14as well as the perspective transformations and image transforms needed to properly augment the first display14.

Rather than an open-ended search for suitable candidates for augmentation, in an embodiment of system10, the second computer18may be configured to recognize only a pre-defined set of visual markers28which allow for several different modes for identifying augmentation candidates. Suitable example uses for which the visual markers28may be used by the second computer18include, but are not limited to: a) specifying the data, images or display elements on the first display14which should be augmented; b) providing a universal resource identifier (URI) or other identification code for retrieval of specific augmentation data; c) providing a set of metadata or search strings which can be used by the second computer18to find augmentations on an AR datastore; d) identifying where on the second display20augmentations should appear as overlays to the first display14; e) identifying individual first displays14by a unique identifier; f) facilitating accurate tracking of the first display14surface, orientation, and physical characteristics (e.g. skew and perspective distortion); g) encoding of primary data22content (such as metadata or description of images, summarizations or full-text of text documents) eliminating the need for OCR; h) facilitating discovery of each first display14in the captured image; and i) facilitating accurate modeling of each first display14's optical characteristics (the visual marker having a known pattern, size, scale, and color characteristics from which differential measurements can be learned).

Differential measurement using a known marker28can allow each of the screen color characteristics, ambient lighting conditions and effects, and physical transform characteristics (such as skew) to all be accounted for using one or more visual markers of known characteristics. In this way the visual markers28may be used as a calibration mechanism. The visual markers28do not need to be shown within the active region of the first display14, but may also be placed adjacent to the first display14or even surrounding the first display14. This may facilitate accurate tracking of display characteristics or may facilitate first display14discovery. The visual markers28displayed on the first display14do not need to be fixed, but might also convey information directly using a marker which is time-coded as well as spatially coded. The use of time-coded visual markers28also greatly improves the bandwidth of data that can be transmitted from the first display14into the second computer18. In addition, the second computer18may extract from the visual marker28data that identifies the computing context of the first computer12(e.g., operating system, which application/program is running, etc.). The visual marker28may also include, but is not limited to, information regarding user context (e.g., which user is logged in), cyber security context (e.g., whether anti-virus software is up to date, whether the running application is approved for use on the first computer18, etc.), and the sensitivity level of the primary data22.

System10may be used to display information of multiple classifications within the same visual space for an individual user. System10uses AR to enable an individual to carry information from a more restricted system along with them into an unrestricted environment, and to consult this information in the context where it would be most useful. Security may be preserved with embodiments of system10which augment only the individual user's environment, while the second display20remains invisible to others. An embodiment of system10may be configured to recognize only the first display14and not other objects in the physical environment. System10is capable of recognizing the primary data22displayed on the first display14and augmenting the primary data22with additional content, ie., the secondary data24, by displaying the secondary data24on the second display20.

A handheld tablet or phone may be used as the second display20, although this introduces the risk of disclosing the secondary data24to nearby parties. The second display20may even incorporate a display that resides within a contact lens. System10may further be configured to augment other senses of the user. For example, an embodiment of system10may further be configured to automatically process audio (or visual) cues from the environment to provide relevant audio data (classified voice recordings, etc) to the user through headphones or ear buds50, such as those shown inFIG. 5. In general, any environmental cue (visual, aural, temperature, magnetic field, geographic location) could be used by system10to confidentially provide relevant information to the user in any mode of sensing (visual, aural, haptic, olfactory).

FIG. 5is an illustration of an embodiment of the heads-up display device30. This embodiment of the heads-up display device30could provide “action cues” to the user in addition to providing data overlays. Action cues are specialized indicators to prompt the user to take some action in their environment, which would be based on information from the higher sensitivity level. These cues could help users apply restricted or sensitive information in an unrestricted environment, for example by using the aforementioned image processing algorithms and techniques to identify a physical keypad and augmenting the user's field of vision with the keypad access code or directly highlighting the correct keys as the user presses them; or by identifying a password field on the primary information display and displaying the corresponding password on the second display20. In the keypad scenario described above, the first display14may be the keypad (being either a digital display or a physical keypad with physical buttons), and/or the password field (being resident within the graphical user interface of a first display14).

The heads-up display device30may also be configured to provide “warning cues” to the user. Warning cues are specialized indicators to alert the user to potential information domain breaches or potentially incorrect or misleading information contained in the primary data22. One example of this is using existing QR codes or other fiducial markers to associate a relatively lower information domain with the primary data22contained on the first display14; in the event that sensitive information from the higher sensitivity domain has inadvertently leaked from the higher sensitivity level and is being displayed in the first display14, the second computer18would place an indicator of this fact on the second display20in the user's field of vision and alert the user to take further action.

The heads-up display device30may be further configured to use commonly available textual analysis algorithms to extract a dictionary of key phrases and terms from sensitive information that is currently augmenting the user's field of vision, and use commonly available speech recognition algorithms to track topics and phrases under discussion by the user and those around them. In the event that the topic or phrases under discussion include information from a sensitivity level that should not be discussed in the given environment, the heads-up display device30may be configured to alert the user visually and/or audibly about the potential information leakage event and to show the location or source of the information leakage on the second display20.

The heads-up display device30may be configured to allow users with varying information privileges to collaborate safely by identifying which users have access to which information. Using face recognition techniques and badge markers, the heads-up display device30may be configured to identify other collaborators in the same physical space as the user, and to check their information privileges with a central database or directory. As the heads-up display device30augmented the user's workspace with sensitive secondary data24, the heads-up display device30may be configured to provide visual indications to the user of which other people in their workspace could access the same sensitive secondary data24. This has the potential to greatly speed discussions in multi-domain environments, since although sensitive information may not be discussed directly, those working with a particular piece of information would have instant visibility into who else in their workspace was apprised of the same data.

The heads-up display device30may be further configured to use a CV or fiducial marker as a form of encryption key to unlock further information in the environment. There are many possible embodiments in which this approach can be implemented and used. For example, when a marker is embedded into a document, the marker can contain or be used as an encryption key to unlock information or the content of other markers in future documents. When the heads-up display device30is shown a marker containing an encryption key, the second computer18can store the encryption key for later use. When a new document is shown which is cypher-text, the second computer18can decrypt the cypher text on the fly and display the decrypted content on the second display20. The heads-up display device30may have encryption keys preloaded and ready for use in decryption of sensitive information.

In an embodiment of system10, the heads-up display device30may be further configured to recognize markers which contain the fingerprint of the correct encryption key to use for decryption of cypher-text shown on the same page or screen region or otherwise associated with the encryption key. The second computer18may already know the encryption key, and may have many individual encryption keys. Recognition of the maker can allow the second computer18to select the correct encryption key for the context.

In an embodiment of system10, the heads-up display device30may be configured to identify instances of sensitivity escalation when combining multiple documents that individually don't contain sensitive information. However, when a person comes across all these documents and their respective markers (keys), the cumulative information may be sensitive. In government or military information systems the aggregation of unclassified information may result in information which has a higher sensitivity, requiring treatment as sensitive but unclassified information, or may result in requiring a higher classification such as SECRET or TOP SECRET. The heads-up display device30may be configured to inform the user that the combined information is sensitive or classified. In some cases the fact that the information can be combined would itself be sensitive. Additionally, the heads-up display device30may be permitted to unlock additional internal information and to subsequently present the augmented info to the user.

In another embodiment of system10, which is related to the scenario of documents becoming classified due to combinations, the second computer18may be configured to determine the information which has already been viewed by the user in order to determine the content of the secondary data24displayed to the user on the second display20. This generally relates to “need to know”. If a user already has access to Document A and Document B, a rule-based machine reasoning algorithm can determine whether the user's previous ‘need to know’ logically translates to a ‘need to know’ of Document C. That is, because Document C is strictly the combination of A and B, then he/she should logically have a need to know for Document C. This embodiment of system10uses an augmented reality approach to detecting which documents have been seen based on a simple set of basic access controls, and utilizes strong encryption to give access to additional information based on the transitive property of combinations of sensitive information.

FIG. 6is an illustration of an embodiment of system10. The aforementioned screen recognition algorithms, image shape transformations, and optical calibration algorithms can be adapted for the recognition of printed materials52on various materials (such as a plastic picture badge) as well as the first displays14. Printed materials generally contain computer generated fonts and other objects which are commonly shown on computer displays. As such, the methods for recognizing many of the common data elements are similar. Print substrates (commonly thin sheets of paper or plastic) also have identifiable optical characteristics which can be used for efficient discovery of image capture regions. In this way, the second computer18of system10can also be used to differentiate between first displays14and non-display rectangular objects in the environment (such as the printed material52which may include sheets of paper with printed text, picture badges, paper receipts, etc.). One reason that we may desire to recognize other rectangular surfaces is so that we can prevent these regions of the captured image from being considered as a first display14, such as the smart phone54embodiment of the first display14shown inFIG. 6. By calibrating the algorithm for shape and optical characteristics peculiar to various rectangular printed materials52, we can better ensure that surfaces which are detected are actually first displays14. Another application of the printed material detection algorithm allows for augmentations of the printed material52, when the printed material52contains primary data22. The printed material52may also contain optical markers28, which may relate to the same set of indicators as for first displays14(such as the classification or sensitivity of the written text, encryption keys and encryption key fingerprints, etc, search criteria, a universal resource indicator (URI), computing context information, user context information, security context, and the sensitivity level of the primary data). By combining the recognition of first displays14and printed material52the system10is capable of combining information between the two sources of different primary data22. By combining this information, decisions on derivative classification, or of computing transitive classification attributes, can be made by the second computer18. For example, if a user was holding a printed page of a properly marked classified document, the system10could recognize the printed material52, process this material, and then search for the same text within nearby first displays14and/or in the secondary data24stored in the second computer18. In this way the system10can use a physical document to discover potential data leaks between information sensitivity levels.

The second computer18may be configured to combine recognition of markers with other augmented reality capabilities, such as ensuring that an individual should have access to sensitive information. For example, system10could use markers on the badge of an individual to pull up information related to the user's level of access to sensitive information. By combining information stored in the marker and/or combining the marker information with data stored on an authentication server along with face recognition, the heads-up display device30could assist in verifying access. The second computer18may be configured to combine trusted information with facial recognition or other feature recognition algorithms to verify identity. In this case the marker is used to indicate a unique ID number that is associated with the person being verified. This embodiment of system10could also be used to enforce “need to know” rules for access to sensitive information. Once identified, an identified user who is wearing an embodiment of the heads-up display device30can be shown an encryption key marker. Once seen by the heads-up display device30, this encryption key marker can then be used to decrypt information for which the identified user has a “need to know.”

The “need to know” rules may also require that the user receive visual markers28in a particular sequence. Based on the sequence or upon the combination of markers seen by the heads-up display device30, the second computer18can unlock sensitive information appropriately. There are many examples where the order in which documents are viewed is important. Often a user is given access based on a prior judgment. System10may be configured to allow the user access to the secondary data24only after the user reviews the classification guidance or indoctrination requirements. While reviewing the guidance the second computer18perceives markers which allow the user additional access. For example, government facilities which handle classified information often require that employees read procedures on how to handle classified documents before the employee is actually allowed to handle classified material. For example, the heads-up display device30may be configured to not do anything with Document B until it has seen the marker in Document A.

Commercial use of the approach described in the paragraph above would ensure that people read User Terms and Agreements, which limit the company's liabilities. For example, each page of a legal agreement may include a marker and the heads-up display device30may be configured to only unlock or decrypt the secondary data24once the computer18has recognized each marker—indicating that the user has viewed each page of a legal agreement. These markers may indicate the location of encryption keys or contain the encryption keys which would subsequently be used for decrypting sensitive information.

From the above description of the methods and systems for augmenting the display of information having a first sensitivity level with information having a higher sensitivity level, it is manifest that various techniques may be used for implementing the concepts described herein without departing from the scope of the claims. The described embodiments are to be considered in all respects as illustrative and not restrictive. The methods and systems for augmenting the display of information having a first sensitivity level with information having a higher sensitivity level disclosed herein may be practiced in the absence of any element that is not specifically claimed and/or disclosed herein. It should also be understood that the invention as claimed is not limited to the particular embodiments described herein, but is capable of many embodiments without departing from the scope of the claims.