System and Method for Watermarking Data for Tracing Access

A method, computer program product, and computing system for receiving, from a requesting party, a request to access data from a storage device. Identity information associated with the requesting party is determined. A bespoke identity-based watermark is generated for the requesting party. The bespoke identity-based watermark is encoded into the data. The watermarked data is provided to the requesting party.

BACKGROUND

Ambient Cooperative Intelligence (ACI) may be used, e.g., to turn conversational speech and/or video information into transcriptions and formatted reports representing a particular encounter (e.g., an online or in-person meeting between individuals). As such, ACI involves safeguarding access to encounter information that is processed into reports and transcriptions.

In addition to encounter information, data of any kind (e.g., videos, images, photographs, documents, audio recordings, etc.) may be vulnerable to unauthorized access or distribution. Watermarking data provides the ability to determine the source of the data. However, conventional approaches to watermarking data do not follow or trace access of the data.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

As discussed above, speech signals/audio information are recorded and used with various speech processing systems and/or machine learning models. In addition to encounter information, data of any kind (e.g., videos, images, photographs, documents, audio recordings, etc.) may be vulnerable to unauthorized access or distribution. Watermarking data provides the ability to determine the source of the data. However, conventional approaches to watermarking data are unable to trace access of the data.

As will be discussed in greater detail below, implementations of the present disclosure provide a technical solution necessarily rooted in computing technology to watermark data that traces the access of the data by particular individuals or entities. Specifically, the data is encoded with a bespoke identity-based watermark associated with the individual or entity. In this manner, access of the data can be traced to the individual or entity that obtains the data from secure storage.

The Watermarking Process:

As discussed above, speech signals/audio information are recorded and used with various speech processing systems and/or machine learning models. In addition to encounter information, data of any kind (e.g., videos, images, photographs, documents, audio recordings, etc.) may be vulnerable to unauthorized access or distribution. Watermarking data provides the ability to determine the source of the data. However, conventional approaches to watermarking data are unable to trace access of the data from secure storage.

As will be discussed in greater detail below, implementations of the present disclosure watermark data that traces the access of the data by particular individuals or entities. Specifically, the data is encoded with a bespoke identity-based watermark associated with the requesting party. In this manner, unauthorized leaks of data can be detected and deterred. For example, suppose a QDS (quality documentation specialist), or a malicious actor with unauthorized access to the QDS's computer, wanted to leak data, in order to learn medical secrets or to damage a particular entity's reputation. As will be discussed in greater detail below, data requested by the QDS from a particular access server is stored in an encrypted form and only decrypted when requested. At the time of decryption, the data is encoded with a bespoke watermark that identifies the requesting entity (e.g., the QDS, in this example). When requests for data are managed in this way, no one, with either legitimate or illegitimate access to the data can get copies of the data without a watermark. This is because the access server has the ability to decrypt the original data. As such, any entity trying to read the data without using the access server will only see encrypted data. Only the access server is able to decrypt the data and will only do so for an authorized request in which it will encode the bespoke watermark in the data it provides to the requesting entity.

Referring toFIGS.1-5, watermarking process10receives100, from a requesting party, a request to access data from a storage device. Identity information associated with the requesting party is determined102. A bespoke identity-based watermark is generated104for the requesting party. The bespoke identity-based watermark is encoded106into the data. The watermarked data is provided108to the requesting party.

In some implementations, watermarking process10receives100, from a requesting party, a request to access data from a storage device. For example, a requesting party is any individual, entity, or device that requests access to the data. In one example, a user desires to obtain information from a secure storage device. In this example, the user, via a client electronic device, provides a request to watermarking process10. In some implementations, an access server selectively provides access to the data. For example and referring also toFIG.2, watermarking process10(e.g., via access server200) receives100a request (e.g., request202) to access data (e.g., data204) from a storage device (e.g., storage device206). In some implementations, access server200is a server or server component configured to manage remote access to data stored on one or more storage devices (e.g., storage device206).

In some implementations, access server200is communicatively coupled to various client electronic devices using various networks and network protocols. Storage device206is communicatively coupled via various known networks and network protocols or may be a portion of access server200. In some implementations and as will be discussed in greater detail below, access server200is a computer system as shown inFIG.5.

In some implementations, the request (e.g., request202) may or may not include any reference to the storage device (e.g., storage device206). In one example, a requesting party (e.g., requesting party208including user210via client electronic device212) provides a general request or query for data204without knowing the location of data204in storage device206. In another example, the requesting party (e.g., requesting party214) knows the location of data204within storage device206and provides a request (e.g., request216) specifically for data204stored within storage device206.

In some implementations, watermarking process10receives100requests from multiple requesting parties. Referring again to the example ofFIG.2and in some implementations, watermarking process10receives100a plurality of requests (e.g., requests202,216) for data (e.g., data204) from a plurality of requesting parties (e.g., requesting parties208,214, where requesting party214includes user218using client electronic device220).

In some implementations, the data includes audio data/audio information. For example and as discussed above, data204represents a portion of audio encounter information. In one example, the data includes automated speech recognition (ASR) audio information. Suppose a conversation is recorded between a doctor and a patient. In this example, the recorded audio information is captured for preserving the content of the conversation (e.g., content of the patient's appointment with the doctor) and for processing into a transcription using an ASR system. In this example, watermarking process10receives ASR audio information for storage in a storage device. For example, watermarking process10receives ASR audio information from an audio recording system and/or from another computing device for processing with an ASR system. In this example, watermarking process10receives ASR audio information for temporary storage (e.g., to process the ASR audio information immediately), for storage until subsequent ASR processing or training (e.g., retaining in storage for future training or processing), and/or for long term storage (e.g., preserving the ASR audio information). As will be discussed in greater detail below, watermarking process10encodes a bespoke identity-based watermark into the ASR audio information at the time of access by a requesting party. In this manner, the ASR audio information is traceable to the requesting party that initially obtains the data from secure storage.

In some implementations, watermarking process10determines102identity information associated with the requesting party. In one example, when accessing access server200, each requesting party (e.g., requesting parties208,214) provides access credentials (e.g., a user name and password) that uniquely identify and authenticate a requesting party for accessing particular data. The access credentials are provided with each request and/or to initiate a session of access communication between the requesting party and access server. During a session, once a requesting party is authenticated, the requesting party accesses the access server without having to re-authenticate for a predefined amount of time (e.g., after 30 minutes of no activity between requesting party and access server). It will be appreciated that various known approaches are used to authenticate a particular requesting party for accessing data using the access server.

Continuing with the example ofFIG.2, suppose watermarking process10receives100a request (e.g., request202) to access data204from requesting party208. In one example, watermarking process10determines102identification information associated with requesting party208using identification/authentication information (e.g., a user name and password) provided in request202. In another example, watermarking process10determines102identification information associated with requesting party208by verifying session information for request202from requesting party208. As such, it will be appreciated that watermarking process10determines102identity information associated with a requesting party in various ways within the scope of the present disclosure.

In some implementations, watermarking process10generates104a bespoke identity-based watermark for the requesting party. Generally, a watermark is metadata that typically describes data origination information (i.e., creator of the data, owner of the data, device(s) used to generate the data (e.g., microphone used to generate audio information), application(s) used to generate the data, timestamp information, identification information for personal information within the data (e.g., audio information with speaker identity information), etc.). In some implementations, this content or payload is encoded in the watermark.

A bespoke identity-based watermark is a watermark that uniquely identifies a particular requesting party from a plurality of requesting parties or potential requesting parties. In some implementations, a bespoke identity-based watermark includes information identifying a requesting party. For example, the bespoke identify-based watermark includes information concerning the request (e.g., timestamp of request, user credentials identified, etc.). In one example and as will be discussed in greater detail below, the bespoke identity-based watermark includes an authorization chain of requesting entities from the time the data is initially stored in the access server. In another example, the bespoke identity-based watermark also includes other metadata (e.g., data origination, timestamps, devices used to generate the data, etc.). In some implementations, the same bespoke identity-based watermark is used to encode any and all data requested by a specific requesting party. In this manner, a single bespoke identity-based watermark traces any and all data accessed from the access server by a particular requesting party. Referring again to the example ofFIG.2and in response to determining102that request202for data204is associated with requesting party208, watermarking process10generates104a bespoke identity-based watermark222for requesting party208.

Referring also to the example ofFIG.3, suppose that watermarking process10receives100request216for data204and that watermarking process10determines102the identification information associated with requesting party214. In this example, watermarking process10generates104a bespoke identity-based watermark (e.g., bespoke identity-based watermark300) for requesting party214. As shown inFIGS.2-3, while each requesting party requests the same data (e.g., data204), watermarking process10generates bespoke identity-based watermarks (e.g., bespoke identity-based watermarks222,300) for each requesting party. In this manner, access to data204is traceable to receiving parties208,214using bespoke identity-based watermarks222,300, respectively.

In some implementations, generating104a bespoke identity-based watermark for the requesting party includes generating110a representation of the identification information associated with the requesting party. A representation of the identification information is a symbol or group of symbols that maps to the identification information. For example, watermarking process10generates104a representation that maps to identification information associated with the requesting party. In one example, watermarking process10uses a hash function to compress the size of the payload of the bespoke identity-based watermark. A hash function maps data of arbitrary size to fixed-size values. The values returned by a hash function are called hash values, hash codes, digests, or hashes. The values are used to index a fixed-size table called a hash table. In this example, watermarking process10generates a bespoke identity-based watermark by generating a hash of the payload including the identification information associated with the requesting party. As discussed above, the payload includes any additional information with the identification information. While an example of a hash has been described as a representation of the identification information, it will be appreciated that watermarking process10generates various types of representations of the identification information within the scope of the present disclosure. For example, watermarking process10uses a lookup table with particular entries that map to specific identification information. In this manner, the payload size of the encoded watermark is reduced from the identification information to a representation that maps to the identification information.

In one example, suppose watermarking process10receives100requests202,216from receiving parties208,214, respectively. In this example, watermarking process10determines the identification information for each requesting party and generates110a representation (e.g., a hash) of the identification information. The identification information and any other payload content for requesting party208is stored in a representation mapping table (e.g., representation mapping table224) and bespoke identity-based watermark222is generated with the representation (e.g., hash or hash value) corresponding to the entry in representation mapping table. Similarly, the identification information and any other payload content for requesting party214is stored in representation mapping table224and bespoke identity-based watermark300is generated with the hash or hash value corresponding to the particular entry in hash table. A representation mapping table (e.g., representation mapping table224) is a data structure or database with a plurality of representations mapping to a plurality of identification information. By using a representation to represent the identification information, the size constraints associated with bespoke identity-based watermarks are reduced.

In some implementations and in response to determining102the identification information associated with the requesting party, watermarking process10obtains the requested data from the storage device. For example, suppose that watermarking process10is unable to identify or authenticate a requesting party when processing a request. In this example, watermarking process10responds to the request with an alert or error messaging inviting the requesting party to resolve the issue and to resubmit the request for data. In another example, suppose that watermarking process10determines102the identification information associated with the requesting party. In this example, watermarking process10obtains the requested data from the storage device for processing on the access server. Referring again to the example ofFIG.2, suppose that watermarking process10determines102the identification information associated with requesting party208(e.g., requesting party208includes user210using client electronic device212). In this example, watermarking process10obtains data204from storage device206for processing using access server200.

In some implementations, watermarking process10decrypts112the data from the storage device using a private key corresponding to the public key. For example and in some implementations, watermarking process10secures data204within storage device206by encrypting the data using an encryption key. As is known in the art, private/public key pairs are a type of asymmetric cryptography, which is a class of cryptographic protocols that are based upon algorithms that require two separate keys (e.g., a private key and a public key), one of which is publicly available (e.g., public key226) and one of which is secret (e.g., private key228). Although these keys (e.g., keys226,228) are different, keys226,228are mathematically linked and complimentary with respect to each other. For example, a public key of an intended recipient is used to encrypt data that is then provided to the recipient in an encrypted form. Since this data was encrypted using the public key, the only way to decrypt the data is using the related private key (which is held in secret by the recipient). Accordingly, only the recipient decrypts the encrypted data. Therefore and if encryption key226is an asymmetric public encryption key of a private/public key pair, the only way that data204is decrypted is through the use of asymmetric private encryption key228, which is stored on access server200.

Specifically and with respect to private/public key pair systems, the private key generally may not be computed in any reasonable amount of time from the public key (e.g., using today's technology and equipment, it may take centuries to determine a private key from its corresponding public key). However and given infinite time and resources, the private key may be mathematically determined from the public key.

Additionally and in some private/public key pair systems, neither the public key nor the private key may be computed in a reasonable amount of time from the other key. However and in other private/public key pair systems, the relationship may be asymmetric and the public key may be computed in a reasonable amount of time from the private key. For this disclosure and without loss of generality, the “private key” is intended to be interpreted as the key that may not be computed from the other in a reasonable amount of time. If both of the keys in a private/public key pair system have this property, then either of the keys may be designated as the “private key” and the other key may be designated the “public key”.

In some implementations, watermarking process10encrypts data204using public key226upon storage within storage device206. In this manner, data204is secure from unauthorized access until data204is decrypted using private key228. Watermarking process10decrypts112data204to allow for the encoding of the bespoke identity-based watermark (e.g., bespoke identity-based watermark222) into data204. As shown inFIG.2, watermarking process10decrypts112data204using private key228to generate decrypted data230.

In some implementations, watermarking process10encodes106the bespoke identity-based watermark into the data. Encoding106the bespoke identity-based watermark into the data includes adding or applying the bespoke identity-based watermark into the data. For example, suppose that data204is an audio file. In this example, watermarking process10encodes104the bespoke identity-based watermark as an inaudible audio signal (e.g., audio representation in the audio file that is outside of human hearing). In another example, the bespoke identity-based watermark is encoded in the noise or other non-speech portions of the audio file. In the context of a an image file, watermarking process10encodes the bespoke identity-based watermark as one or more pixels or other visual markers in particular portions of the image file. In the example of a video file, watermarking process10encodes the bespoke identity-based watermark in the combination of an audio signal and one or more pixels or other visual markers in particular portions of the video file. In this manner, watermarking process10encodes or adds the bespoke identity-based watermark into the data using any of the various known approaches such that the bespoke identity-based watermark is decodable from the watermarked data. In some implementations, watermarking process10uses different watermarking techniques for different types of requested data.

In some implementations, encoding106the bespoke identity-based watermark into the data includes encoding114the representation of the identification information associated with the requesting party into the data. As discussed above, suppose watermarking process10generates104bespoke identity-based watermark222for requesting party208and bespoke identity-based watermark300for requesting party214. For example, suppose that requesting party208requests access to data204at 12:00 PM on 7/1/2022. Watermarking process10saves this identification information in representation mapping table224and generates104bespoke identity-based watermark222(e.g., hash value: 0x049a3f364) mapping to the identification information for requesting party208. Now suppose that requesting party214requests access to data204at 3:22 PM on 7/26/2022. Watermarking process10stores this identification information in representation mapping table224and generates104bespoke identity-based watermark300(e.g., hash value: 0x36098ab0c) mapping to the identification information for requesting party214. In this example, each bespoke identity-based watermark is a hash value corresponding to particular entries of representation mapping table224including the identification information and any other payload for each requesting party's watermark.

As discussed above, suppose data204is audio information or an audio file (e.g., ASR training data). In this example, watermarking process10encodes a hash value corresponding to the particular entries of the hash table for a particular bespoke identity-based watermark by generating an audio representation of the hash value and adding the audio representation to the decrypted audio information. As will be discussed in greater detail below, when a leaked document is detected with a watermark for which the payload is a hash, watermarking process10looks up the hash in the representation mapping table (e.g., representation mapping table224) to identify the requesting party. For example, suppose that watermarking process10decodes the hash 0x049a3f364, then requesting party208leaked data204. In another example, suppose that watermarking process10decodes the hash 0x36098ab0c, then requesting party214leaked data204.

In some implementations, encoding114the representation of the identification information associated with the requesting party into the data includes encoding each representation in particular portions of the data. For example, watermarking process10defines particular encoding parameters (e.g., frequencies of an audio signal to encode a particular representation in, phase parameters to encode the representation in, etc.). In this manner, encoding each representation of the identification information includes encoding each representation using particular encoding parameters defined for a specific representation. In this example, the diverse encoding parameters for each representation reduces the likelihood that the encoded watermarks are detected and removed.

In some implementations, watermarking process10provides108the watermarked data to the requesting party. Referring again toFIG.2, suppose that bespoke identity-based watermark222is generated for requesting party208and that decrypted data230is obtained. In this example, watermarking process10encodes106bespoke identity-based watermark222into decrypted data230to generate watermarked data232. Similarly and as shown inFIG.3, suppose that watermarking process10generates104bespoke identity-based watermark300for requesting party214. In this example, watermarking process10encodes106bespoke identity-based watermark300into decrypted data230to generate watermarked data302. In this manner, the same data (e.g., decrypted data230) is watermarked with bespoke identity-based watermarks generated for each requesting party (e.g., receiving parties208,214). Watermarking process10provides108the watermarked data (e.g., watermarked data232,302) to the requesting party (e.g., requesting parties208,214, respectively).

In some implementations, watermarking process10decodes116the bespoke identity-based watermark to identify the requesting party. Decoding116the bespoke identity-based watermark includes processing the bespoke identity-based watermark from the suspect watermarked data and attempting to read the watermark from the data. For example, at any point in time after watermarking process10provides108the watermarked data (e.g., watermarked data232,302) to requesting parties208,214, watermarking process10identifies a copy of watermarked data (or data believed to be watermarked data) outside of the approved access path or circle. Accordingly, watermarking process10can decode116the bespoke identity-based watermark to trace the unauthorized access of the data back to the requesting party that requested the data from secure storage initially.

Referring again toFIG.3, suppose that watermarking process10identifies data204from an unauthorized source and/or in an unprotected environment. For example, suppose that watermarked data232is found on a public website or publicly accessible storage system. At this point in time, it is unclear how this data became accessible. In this example, watermarking process10decodes116the bespoke identity-based watermark from watermarked data232. Specifically, suppose that watermarking process10decodes116bespoke identity-based watermark222associated with requesting party208. In this example, watermarking process10engages in a remedial action (e.g., generate an alert or notification alerting a storage device administrator, the original owner of the particular data, the requesting party, take a data security action (e.g., prevent subsequent data access to the identified requesting party, revoke access, initiate an investigation, etc.). By decoding116bespoke identity-based watermark222, watermarking process10is able to trace the source of a potential data breach by identifying the requesting party that obtained the data from secure storage. In this example, despite receiving parties208,214both requesting data204, watermarking process10is able to trace the exposure of data232back to requesting party208using bespoke identity-based watermark222.

In some implementations, the bespoke identity-based watermark allows for tracing data's chain of access. For example and referring also toFIG.4, suppose that requesting party208is provided with watermarked data232in response to a request for data204. Further suppose that requesting party214requests access to data204and that watermarking process10provides watermarked data302. Now suppose that requesting party400(e.g., user402with client electronic device404) requests data204and that watermarking process10provides watermarked data406. In this example, suppose that data204is leaked to a public resource. In this example, watermarking process10uses the bespoke identity-based watermark to generate an audit trail for data204. Suppose that inspection of data204reveals watermarks222and300but does not find a watermark identifying requesting party400. In this example, because requesting party214is the last accessing party referenced by the bespoke identity-based watermarks, watermarking process10investigates a leak of data204beginning with requesting party214. Accordingly and in some implementations, watermarking process10uses a plurality of bespoke identity-based watermarks to trace data access and potential data leaks.

Referring also toFIG.5, there is shown watermarking process10. Watermarking process10may be implemented as a server-side process, a client-side process, or a hybrid server-side/client-side process. For example, watermarking process10may be implemented as a purely server-side process via watermarking process10s. Alternatively, watermarking process10may be implemented as a purely client-side process via one or more of watermarking process10c1, watermarking process10c2, watermarking process10c3, and watermarking process10c4. Alternatively still, watermarking process10may be implemented as a hybrid server-side/client-side process via watermarking process10sin combination with one or more of watermarking process10c1, watermarking process10c2, watermarking process10c3, and watermarking process10c4.

Accordingly, watermarking process10as used in this disclosure may include any combination of watermarking process10s, watermarking process10c1, watermarking process10c2, watermarking process10c3, and watermarking process10c4.

Watermarking process10smay be a server application and may reside on and may be executed by a computer system500, which may be connected to network502(e.g., the Internet or a local area network). Computer system500may include various components, examples of which may include but are not limited to: a personal computer, a server computer, a series of server computers, a mini computer, a mainframe computer, one or more Network Attached Storage (NAS) systems, one or more Storage Area Network (SAN) systems, one or more Platform as a Service (PaaS) systems, one or more Infrastructure as a Service (IaaS) systems, one or more Software as a Service (SaaS) systems, a cloud-based computational system, and a cloud-based storage platform.

A SAN includes one or more of a personal computer, a server computer, a series of server computers, a mini computer, a mainframe computer, a RAID device and a NAS system. The various components of computer system500may execute one or more operating systems.

The instruction sets and subroutines of watermarking process10s, which may be stored on storage device504coupled to computer system500, may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included within computer system500. Examples of storage device504may include but are not limited to: a hard disk drive; a RAID device; a random access memory (RAM); a read-only memory (ROM); and all forms of flash memory storage devices.

Various IO requests (e.g., IO request508) may be sent from watermarking process10s, watermarking process10c1, watermarking process10c2, watermarking process10c3and/or watermarking process10c4to computer system500. Examples of IO request508may include but are not limited to data write requests (i.e., a request that content be written to computer system500) and data read requests (i.e., a request that content be read from computer system500).

The instruction sets and subroutines of watermarking process10c1, watermarking process10c2, watermarking process10c3and/or watermarking process10c4, which may be stored on storage devices510,512,514,516(respectively) coupled to client electronic devices212,518,520,522(respectively), may be executed by one or more processors (not shown) and one or more memory architectures (not shown) incorporated into client electronic devices212,518,520,522(respectively). Storage devices510,512,514,516may include but are not limited to: hard disk drives; optical drives; RAID devices; random access memories (RAM); read-only memories (ROM), and all forms of flash memory storage devices. Examples of client electronic devices212,518,520,522may include, but are not limited to, personal computing device212(e.g., a smart phone, a personal digital assistant, a laptop computer, a notebook computer, and a desktop computer), audio input device518(e.g., a handheld microphone, a lapel microphone, an embedded microphone (such as those embedded within eyeglasses, smart phones, tablet computers and/or watches) and an audio recording device), display device520(e.g., a tablet computer, a computer monitor, and a smart television), machine vision input device522(e.g., an RGB imaging system, an infrared imaging system, an ultraviolet imaging system, a laser imaging system, a SONAR imaging system, a RADAR imaging system, and a thermal imaging system), a hybrid device (e.g., a single device that includes the functionality of one or more of the above-references devices; not shown), an audio rendering device (e.g., a speaker system, a headphone system, or an earbud system; not shown), various medical devices (e.g., medical imaging equipment, heart monitoring machines, body weight scales, body temperature thermometers, and blood pressure machines; not shown), and a dedicated network device (not shown).

Users210,524,526,528may access computer system500directly through network502or through secondary network506. Further, computer system500may be connected to network502through secondary network506, as illustrated with link line530.

The various client electronic devices (e.g., client electronic devices212,518,520,522) may be directly or indirectly coupled to network502(or network506). For example, personal computing device212is shown directly coupled to network502via a hardwired network connection. Further, machine vision input device522is shown directly coupled to network506via a hardwired network connection. Audio input device518is shown wirelessly coupled to network502via wireless communication channel532established between audio input device520and wireless access point (i.e., WAP)534, which is shown directly coupled to network502. WAP534may be, for example, an IEEE 802.11a, 802.11b, 802.11g, 802.11n, Wi-Fi, and/or Bluetooth™ device that is capable of establishing wireless communication channel532between audio input device518and WAP534. Display device520is shown wirelessly coupled to network502via wireless communication channel536established between display device522and WAP538, which is shown directly coupled to network502.

The various client electronic devices (e.g., client electronic devices212,518,520,522) may each execute an operating system, wherein the combination of the various client electronic devices (e.g., client electronic devices212,518,520,522) and computer system500may form modular system540.