Office Action 2: Notice of Allowance (2024-08-23)

CLAIMS:
This listing of claims SVG {EMBED HERE} 0.3332.15 and listings, of claims in the application. The following listing provides the amended claims with deleted material crossed out and new material underlined to show the changes made.

1\. (Currently Amended) A method comprising:receiving surgical video captured by an endoscope that is inserted inside a patient's body during a surgical procedure performed within an operating room (OR);
detecting an out-of-body (OOB) event in a video segment of the surgical video thatoccurs when the endoscope was taken out of a patient's body while the endoscope is still onjby detecting, using an image processing algorithm, 1) a field of view of the endoscope that transitions from within the patient's body to outside the patient's body in a first set of video frames of the surgical video as beginning phase of the OOB event and 2) a subsequent field of view of the endoscope that transitions from outside the patient's body to inside the patient's body in a second set of video frames of the surgical video as an ending phase of the OOB event, anddetermining that a time interval between the beginning phase and the ending phase is within a threshold period of time; and
responsive to detecting the OOB event, de-identifying personally-identifiable information (PII) embedded within the video segment.
2\. (Currently Amended) The method of claim 1, wherein the video segment comprises a plurality of video frames that make up the OOB event, wherein the PII that is de-identified is within a portion but not all of the plurality of video frames .
3\. (Original) The method of claim 1, wherein the PII comprises textual information or faces associated with a patient or a surgical staff of the surgical procedure, wherein de-identifying the PII comprises blurring or editing out the textual information or the faces from the video segment.
4\. (Currently Amended) The method of claim 1, wherein the surgical video comprises a plurality of video frames that comprise the first and second sets of video frames and the video

segment comprises a sequence of one or more video frames of the plurality of video frames, wherein de-identifying the PII comprises blacking out each video frame of the sequence of one or more video frames that are between the first and second sets of video frames.

5\. (Currently Amended) The method of claim 1further comprising identifying the video segment of the surgical video based on the surgical procedure, wherein the detecting of the OOB event is performed responsive to identifying the video segment .

6\. (Currently Amended) The method of claim \[\[5\]\]1, wherein the image processing algorithm is a machine-learning (ML) based OOB-event detection model that is trained to detect OOB events based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.

7\. (Currently Amended) The method of claim 5, wherein the surgical video comprises one or more de-identified video segments, wherein identifying the video segment comprises:determining that the video segment has a high-probability of having the OOB event based on the surgical video; and
 determining that the video segment is not one of the one or more de- identifiedvideo segments.

8\. (Currently Amended) An apparatus comprising:at least one processor; and
memory having stored therein instructions that when executed by the at least one processor causes the apparatus to:
receive surgical video captured by an endoscope that is inserted inside a patient's body during a surgical procedure performed within an operating room (OR);

detect an out-of-body (OOB) event in a video segment of the surgical video;
 thatoccurs when the endoscope was taken out of a patient's body while the endoscope is still on by\_ detecting, using an image processing algorithm, 1) a field of view of the endoscope that transitions from within the patient's body to outside the patient's body in a first set of video frames of the surgical video as beginning phase of the OOB event and 2) a subsequent field of view of the endoscope that transitions from outside the patient's body to inside the patient's body in a second set of video frames of the surgical video as an ending phase of the OOB event, anddetermining that a time interval between the beginning phase and the ending phase is within a threshold period of time; and
responsive to detecting the OOB event, de-identify personally-identifiable information (PII) embedded within the video segment.

9\. (Currently Amended) The apparatus of claim 8, wherein the surgical video comprises a data file with raw video data previously captured by the endoscope during a past surgical procedure performed within the OR .

10\. (Currently Amended) The apparatus of claim 8, wherein the PII comprises textual information or faces associated with a patient or a surgical staff of the surgical procedure, wherein the instructions to de-identify the PII comprises instructions to blur or edit out the textual information or the faces from the video segment.

11\. (Currently Amended) The apparatus of claim 8, wherein the surgical video comprises a plurality of video frames that comprise the first and second sets of video frames and the video segment comprises a sequence of one or more video frames of the plurality of video frames, wherein the instructions to de-identify the PII comprises instructions to black out each video frame of the sequence of one or more video frames that are between the first and second sets of video frames.
12\. (Currently Amended) The apparatus of claim 8, wherein the memory comprises further instructions to identify the video segment of the surgical video based on Attorney Docket No. 210231.P1074USC2 4 the surgical procedure, wherein the detecting of the OOB event is performed responsive to an identification of the video segment ' .

13\. (Currently Amended) The apparatus of claim \[\[12\]\]8, wherein the image processing algorithm is a machine-learning (ML) based OOB-event detection model that is trained to detect OOB events based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.

14\. (Currently Amended) The apparatus of claim 12, wherein he surgical video comprises one or more de-identified video segments, wherein the instructions to identify the video segment comprises instructions to:determine that the video segment has a high-probability of having the OOB event based on he surgical video;
and determine hat the video segment is not one of the one or more de-identified video segments .

15\. (Currently Amended) A non-transitory machine-readable medium having instructions that when executed by at least one processor causes the at least one processor to:receive surgical video captured by an endoscope that is inserted inside a patient's body during a surgical procedure performed within an operating room (OR);
detect an out-of-body (OOB) event in a video segment of the surgical video that occurs when the endoscope was taken out of a patient's body while the endoscope is still on by detecting, using an image processing algorithm, 1) a field of view of the endoscope that transitions from within the patient's body to outside the patient's body in a first set of video frames of the surgical video as beginning phase of the OOB event and 2) a subsequent field of

view of the endoscope that transitions from outside the patient's body to inside the patient's body in a second set of video frames of the surgical video as an ending phase of the OOB event, anddetermining that a time interval between the beginning phase and the ending phase is within a threshold period of time; and
responsive to detecting the OOB event, de-identify personally-identifiable information (PII) embedded within the video segment.

16\. (Currently Amended) The non-transitory machine-readable medium of claim 15, wherein the PII comprises textual information or faces associated with a patient or a surgical staff of the surgical procedure, wherein the instructions to de-identify the PII comprises instructions to blur or edit out the textual information or the faces from the video segment.

17\. (Currently Amended) The non-transitory machine-readable medium of claim 15, wherein the surgical video comprises a plurality of video frames that include the first and second sets of video frames and the video segment comprises a sequence of one or more video frames of the plurality of video frames, wherein the instructions to de-identify the PII comprises instructions to black out each video frame of the sequence of one or more video frames that are between the first and second sets of video frames.

18\. (Currently Amended) The non-transitory machine-readable medium of claim 15comprises further instructions to identify the video segment of the surgical video based on the surgical procedure, wherein the detecting of the OOB event is performed responsive to identifying the video segment '.

19\. (Currently Amended) The non-transitory machine-readable medium of claim \[\[18\]\],15 wherein the image processing algorithm is a machine-learning (ML) based OOB-event detection model that is trained to detect OOB events based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.

20\. (Currently Amended) The non-transitory machine-readable medium of claim 18, wherein the surgical video comprises one or more de-identified video segments, wherein the instructions to identify the video segment comprises instructions to:determine that the video segment has a high-probability of having the OOB event based on the surgical video; and
determine that the ideo segment is not one of the one or more de-identified video segments .

OFFICE ACTION:
DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to communication filed on June 13, 2024.
Status of claims within the present application:
Claims 1 – 20 are pending.
Claims 1 – 2 and 4 – 20 are amended.
Terminal Disclaimer
The terminal disclaimer filed on August 12, 2024 disclaiming the terminal portion of any patent granted on this application which would extend beyond the expiration date of US 11,080,424 B2 has been reviewed and is accepted.  The terminal disclaimer has been recorded.
Response to Arguments
Regarding Claims 1 – 20 that were rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 – 19 of U.S. Patent No. 11080424 B2, applicant’s argument, see page [8], filed on June 13, 2024, have been fully considered and persuasive, therefore, the rejection is withdrawn.
Claims 1 – 20 that were rejected under 35 U.S.C. 103 as being unpatentable over US 20180110398 A1 to Schwartz et al., (hereinafter, “Schwartz”) in view of US 20200285771 A1 to Dey et al., (hereinafter, “Dey”), applicant’s argument, see page [8 – 11], filed on June 13, 2024, have been fully considered and persuasive, therefore, the rejection is withdrawn.
Allowable Subject Matter
Claims 1 – 20 are allowed. The following is an examiner’s statement of reasons for allowance: the following prior arts were yielded during examination of the claims filed on June 13, 2024 in response to office action mailed on March 13, 2024. They do not explicitly teach “occurs when the endoscope was taken out of a patient's body while the endoscope is still on by:  detecting, using an image processing algorithm, 1) a field of view of the endoscope that transitions from within the patient's body to outside the patient's body in a first set of video frames of the surgical video as beginning phase of the OOB event and 2) a subsequent field of view of the endoscope that transitions from outside the patient's body to inside the patient's body in a second set of video frames of the surgical video as an ending phase of the OOB event, and determining that a time interval between the beginning phase and the ending phase is within a threshold period of time;”, but they are in general realm of applicant’s field or endeavor:
Schwartz et al. [US 20180110398 A1]: This is considered closet prior art to the present application that has methodology and technology for automatically recording a video output signal from an endoscopy device. Video image data is received at a video capture device, and transmitted with associated metadata to a server. Prior to transmission, the video capture device analyzes the video image data to identify a recording start indicator as a function of predetermined threshold conditions. Once the indicator is identified, the device sends recording start timestamp to a backend server. The server indexes and stores the recorded video data as a function of the associated metadata in a database, and also records the recording the recording start timestamp data in the database. The device is further able to identify a recording stop indicator, and prepare recording stop timestamp data for transmission to the server.
Schwartz does disclose a video capture system 104 of the system 100 receives a video image from a video input source 102 that originates from an endoscopic imaging system 101 for obtaining a video image from an endoscope positioned in an internal body cavity of a patient. The system 101 may comprise any of a number of commercially-available endoscopic imaging systems, for example including the PINPOINT endoscopic fluorescence imaging system available from NOVADAQ of Bonita Springs, Fla. Imaging may be directed to any of a variety of endoscopic devices, including but not limited to arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes, uteroscopes and colonoscopes.
Dey et al. [US 20200285771 A1]: This prior art discloses methodology for removing personally identifiable information (PII) from medical data and ensuring strict compliance with privacy laws and regulations. The PII removal tool removes, deletes or redacts sensitive personally identifiable information, which may comprise any health care data that may be linked to any individual. The system and method is preferably applied to medical data, images, video, or other electronic media captured from medical procedures, endoscopes, or arthroscopic surgery. Artificial intelligence powered software and models are applied to the medical data for the detection of PII comprising patient names, medical history, faces, account numbers, digital identities, biometric data, or other sensitive personal information which may be linked to an individual. Medical data is re-written without PII and the system generates metadata describing the deleted PII. Privacy law compliant medical data is normalized, stored hierarchically, and transmitted to third party storage systems for analysis with software models for medical insights, anatomical recognition, surgical metrics, patient outcome, diagnosis and medical payment and billing.
Dey does disclose specific artificial intelligence (AI) models may preferably detect, for example, when the medical images or videos of a surgery, are outside the body and recording personally identifiable information (PII), such as people's faces. the system AI models mark these segments containing PII, remove them from the medical video or image files, and then re-assemble the medical data files, images and video, and outputs the data for the next processing stage.
However, none of the prior arts of record independently or in-combination discloses all the limitation of the independent claims 1, 8, and 15 as recited in the amended set of claims being examined.
Therefore, the independent claims are allowable over the prior arts of record. The dependent claims being definite, further limiting, and fully enabled by the specification are also allowed by virtue of their dependence on the independent claims.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be labeled “Comments on Statement of Reasons for Allowance.”
In most cases, the examiner's actions and the applicant's replies make evident the reasons for allowance, satisfying the record as a whole proviso of the rule. This is particularly true when applicant fully complies with 37 CFR 1.111 (b) and (c) and 37 CFR 1.133(b). Thus, where the examiner's actions clearly point out the reasons for rejection and the applicant's reply explicitly presents reasons why claims are patentable over the reference, the reasons for allowance are in all probability evident from the record and no statement should be necessary. Conversely, where the record is not explicit as to reasons, but allowance is in order, then a logical extension of 37 CFR 1.111 and 1.133 would dictate that the examiner should make reasons of record and such reasons should be specific.
Conclusion

APPLICANT REBUTTAL: REMARKS
In response to the above-identified Office Action, Applicant has amended claims 1, 2, and 4-20. Applicant has not canceled or added any new claims. Support for these amendments can be found in the originally filed application. Thus, no new matter has been added. Accordingly, claims 1-20 will be pending in the present application. In view of the remarks presented herein, Applicant requests reconsideration and allowance of the present application.
I.
Applicant's Statement of the Substance of the Interview
Applicant's representative had a telephonic interview with Examiner Pham and Zoubair on June 5, 2024. Applicant's representative Matthew Chavez (Reg. No. 68,318) participated inthe interview. During the interview, Applicant's representative discussed possible claim amendments to the independent claims and proposed remarks in response to the rejections presented in the Office Action. No agreement was reached.
Applicant's representatives did not present any exhibit or conduct any demonstrations. Applicant thanks the Examiners for the telephonic interview. The present Applicant's statement of the substance of the interview is submitted along with the response to the pending Office Action and is therefore timely.
II.
Double Patentin
g
Claims 1-20 stand rejected on the ground of non-statutory obviousness-type double patenting as allegedly being unpatentable over claims 1-19 of U.S. Patent No. 11,080,424. Applicant respectfully requests that the nonstatutory double patenting rejections be held in abeyance until it is determined that the claims are otherwise allowed.
III.
35 U.S.C.
103: Rejection of Claims 1-20
Claims 1-20 are rejected under 35 U.S.C. §103 as being unpatentable over United States Patent Publication 2018/0110398 by Schwartz in view of United States Patent Publication 2020/0285771 by Dey.
A.
Inde
p
endent Claims
Claim 1 recites a method that includes:
receiving surgical video captured by an endoscope that is inserted inside a
patient's body during a surgical procedure performed within an operating room
(OR);
detecting an out-of-body (OOB) event in a video segment of the surgical
video that occurs when the endoscope was taken out of a patient's body while the
endoscope is still on by:
detecting, using an image processing algorithm, 1) afield of view of the endoscope that transitions from within the patient's body to outside the patient's body in a first set of video frames of the surgical video as beginning phase of the OOB event and 2) a subsequent field of view of the endoscope that transitions from outside the patient's body to inside the patient's body in a second set of video
frames of the surgical video as an ending phase of the O0B event, and
determining that a time interval between the beginning phase and the ending phase is within a threshold period of time; and
responsive to detecting the OB event, de-identifying personally- identifiable information (P11) embedded within the video segment.
In other words, the OOB event is detected using an image processing algorithm to detect 1) a beginning phase of the OOB event in a first set of video frames captured by an endoscope and 2) an ending phase of the OOB event in a subsequent set of video frames captured by the endoscope, and determining that a time interval between both phases are within a threshold period of time. Applicant respectfully submits that the relied upon sections of the relied upon references do not disclose or suggest such features.
In rejecting similar features, the Office Action cites paragraphs [0018] and [0019] of Dey. See Office Action, pages 13-16. Applicant respectfully disagrees. The cited portions of Dey describe how a specific Al model detects people's faces that are outside a human body. Nowhere, however, do the cited sections of Dey disclose or suggest detecting an OOB event by 1) using an image processing algorithm to detect beginning and ending phases within different sets of video frames and 2) determining that a time interval between those phases are within a threshold period of time.
In addition, Applicant respectfully submits that the relied upon sections of the relied
upon references do not disclose or suggest detecting the OOB event by determining that a time interval between the beginning phase and the ending phase is within a threshold period of time.
In rejecting similar features, the Office Action appears to assert that paragraph [0019] discloses such features. Applicant respectfully disagrees. The cited portion of Dey describes that medical imaging data may be deconstructed into individual frames for detecting PII on a frame by frame basis, and that the system may label, mark, tag, or code the beginning and ending of segments of video data frames in which PII is present. Thus, Dey is describing tagging segments of video data frames in which PII is present, which is in contrast to the claimed features in which OOB events are detected upon determining that a time interval between the beginning and ending phases is within a threshold period of time.
For at least the foregoing reasons, the cited sections of the relied upon references fail to teach each of the limitations of claim 1.
Applicant respectfully submits that the dependent claims are patentable and non-obvious in light of the relied upon references.
Independent claims 8 and 15 are submitted as not being obvious for at least the same reasons given above in support of independent claim 1.
B.
Dependent Claims
Regarding dependent claim 2, Applicant respectfully submits that the relied upon sections of the relied upon references do not disclose or suggest wherein the video segment comprises a plurality of video frames that make up the O0B event, wherein the PHI that is de- identified is within a portion but not all of the plurality of video frames. Instead, the cited portions of Dey describe identifying PII, such as people's faces, so as to apply additional processing to remove any of the PII. See paragraph [0018]. The identification may include labeling, marking, tagging, or coding video data frames in which the PII is present. See paragraph [0019]. Thus, Dey is only identifying and processing video data frames that include PII, which is in contrast to the claimed features in which a portion of a detected OOB event segment includes PII that is being de-identified.
Any dependent claims not mentioned above are submitted as not being anticipated or obvious, for at least the same reasons given above in support of their base claims, but also for additional reasons not given here in the interest of conciseness.
It should be noted that not all of the assertions made in the Office Action, particularly those with respect to the dependent claims, have been addressed here, in the interest of conciseness. Applicant reserves the right to challenge any of the assertions made in the Office Action by the Examiner, with respect to the relied upon art references and how they would relate to Applicant's claim language, including the right to swear behind or otherwise remove an improper art reference.
In view of the foregoing, it is believed that all claims now pending patentably define the subject invention over the prior art of record and are in condition for allowance and such action is earnestly solicited at the earliest possible date.
CONCLUSION


CITED REFERENCE SUMMARIES: Ref US11080424B2: Claims:
What is claimed is: 1. A computer-implemented method for anonymizing raw surgical procedure videos recorded by a recording device within an operating room (OR), the method comprising: receiving a set of raw surgical videos corresponding to a surgical procedure performed within the OR; merging the set of raw surgical videos to generate a surgical procedure video corresponding to the surgical procedure, wherein the surgical procedure video comprises a set of raw video images; detecting image-based personally-identifiable information embedded in the set of raw video images, which includes detecting a type of image-based personally-identifiable information associated with one or more surgical tools used during the surgical procedure, wherein the set of raw video images includes images of the one or more surgical tools captured inside a patient's body when used to perform the surgical procedure; and when image-based personally-identifiable information is detected, de-identifying the detected image-based personally-identifiable information in the surgical procedure video. 2. The computer-implemented method of claim 1, wherein merging the set of raw surgical videos to generate the surgical procedure video includes: analyzing a set of file names associated with the set of raw surgical videos to determine a correct order with respect to the surgical procedure; and stitching together the set of raw surgical videos based on the determined order. 3. The computer-implemented method of claim 1, wherein detecting the type of image-based personally-identifiable information associated with the one or more surgical tools further includes: detecting personally-identifiable text printed on the one or more surgical tools in the images of the one or more surgical tools; wherein if personally-identifiable text is detected in the images of the one or more surgical tools, de-identifying the detected image-based personally-identifiable information by blurring out or otherwise making the detected text unreadable in the images of the one or more surgical tools. 4. The computer-implemented method of claim 1, wherein the image-based personally-identifiable information further includes: text displayed inside the OR where the surgical procedure was performed, wherein the text was recorded by the recording device during an out-of-body (OOB) event when the recording device was taken out of the patient's body. 5. The computer-implemented method of claim 3, wherein detecting the personally-identifiable text printed on the one or more surgical tools includes: detecting, using a machine-learning-based tool-detection-and-recognition model, a surgical tool within a first subset of the set of raw video images; and processing a portion of the first subset of the set of raw video images containing the detected surgical tool to detect any personally-identifiable text within the portion of the first subset of the set of raw video images. 6. The computer-implemented method of claim 1, wherein the image-based personally-identifiable information further includes a text box inserted into the surgical procedure video to display surgical procedure-related information. 7. The computer-implemented method of claim 6, wherein detecting the text-box in the surgical procedure video includes: detecting, using a machine-learning-based text-box detection model, a text box at or in the vicinity of a predetermined location within a second subset of the set of raw video images; and processing a portion of the second subset of the set of raw video images containing the detected text box to detect any personally-identifiable text within the portion of the second subset of the set of raw video images. 8. The computer-implemented method of claim 1, wherein detecting the image-based personally-identifiable information embedded in the set of raw video images of the surgical procedure video further includes: scanning the set of raw video images to detect a video segment corresponding to an OOB event when the recording device was taken out of a patient's body, wherein personally-identifiable information can be captured by the recording device during the OOB event; and wherein if an OOB event is detected within the set of raw video images, de-identifying the detected image-based personally-identifiable information by automatically blurring out or otherwise edited out each video image in the detected video segment corresponding to the detected OOB event, so that any personally-identifiable information embedded within the detected video segment cannot be identified. 9. The computer-implemented method of claim 8, wherein detecting a video segment corresponding to an OOB event in the set of raw video images includes: detecting, using a machine-learning-based OOB-event detection model, a beginning phase of an OOB event when an endoscope camera is being taken out of a patient's body; detecting, using the machine-learning-based OOB-event detection model, an ending phase of the OOB event when the endoscope camera is being inserted back into the patient's body; and marking a set of video images between the detected beginning phase and the detected ending phase in the set of raw video images as the video segment corresponding to the detected OOB event. 10. The computer-implemented method of claim 9, wherein prior to using the machine-learning-based OOB-event detection model to detect an OOB event, the method further comprises training the OOB-event detection model based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos. 11. The computer-implemented method of claim 1, wherein prior to merging the set of raw surgical videos, the method further includes: processing the received set of raw surgical videos to detect text-based personally-identifiable information embedded in file-structure data associated with the set of raw surgical videos; and if text-based personally-identifiable information is detected, removing or otherwise de-identifying the detected text-based personally-identifiable information from the file-structure data. 12. The computer-implemented method of claim 11, wherein the file-structure data include: file identifiers associated with the set of raw surgical videos; folder identifiers of a file folder containing the set of raw surgical videos; file properties associated with the set of raw surgical videos; and other metadata associated with the set of raw surgical videos. 13. The computer-implemented method of claim 1, wherein after de-identifying the detected image-based personally-identifiable information in the surgical procedure video, the method further includes: performing a random sampling within the de-identified surgical procedure video to randomly select a number of video segments within the de-identified surgical procedure video; and verifying that the randomly selected video segments are free of any personally-identifiable information. 14. The computer-implemented method of claim 1, wherein after de-identifying the detected image-based personally-identifiable information in the surgical procedure video, the method further includes: permanently removing the set of raw surgical videos from a surgical video repository; and replacing the removed raw surgical videos with de-identified surgical procedure video. 15. The computer-implemented method of claim 1, wherein the personally-identifiable information includes both patient identifiable information associate with a patient receiving the surgical procedure and surgical staff identifiable information associated with surgical staff performing the surgical procedure. 16. A system for anonymizing raw surgical procedure videos recorded by a recording device within an operating room (OR), the system comprising: one or more processors; a memory coupled to the one or more processors; and wherein the memory stores a set of instructions that, when executed by the one or more processors, cause the system to: receive a set of raw surgical videos corresponding to a surgical procedure performed within the OR; merge the set of raw surgical videos to generate a surgical procedure video corresponding to the surgical procedure, wherein the surgical procedure video comprises a set of raw video images; detect personally-identifiable information embedded in the set of raw video images of the surgical procedure video, which includes detecting a type of image-based personally-identifiable information associated with one or more surgical tools used during the surgical procedure, wherein the set of raw video images includes images of the one or more surgical tools captured inside a patient's body when used to perform the surgical procedure; and when personally-identifiable information is detected, de-identify the detected personally-identifiable information in the surgical procedure video. 17. The system of claim 16, wherein detecting the type of personally-identifiable information associated with the one or more surgical tools further includes: detecting personally-identifiable text printed on the one or more surgical tools in the images of the one or more surgical tools; and wherein if personally-identifiable text is detected in the images of the one or more surgical tools, de-identifying the detected personally-identifiable information includes blurring out or otherwise making the detected text unreadable in the images of the one or more surgical tools. 18. The system of claim 16, wherein detecting the personally-identifiable information embedded in the set of raw video images of the surgical procedure video further includes scanning the set of raw video images to detect a video segment corresponding to an OOB event when the recording device was taken out of a patient's body where the surgical procedure was performed; and wherein if an OOB event is detected within the set of raw video images, de-identifying the detected personally-identifiable information by automatically blurring out or otherwise edited out each video image in the detected video segment corresponding to the detected OOB event, so that any personally-identifiable information embedded within the detected video segment cannot be identified. 19. The system of claim 16, wherein after de-identifying the detected personally-identifiable information in the surgical procedure video, the system is configured to: permanently remove the set of raw surgical videos from a surgical video repository; and replace the removed raw surgical videos with de-identified surgical procedure video.
A computer-implemented method for anonymizing raw surgical procedure videos recorded by a recording device within an operating room (OR), the method comprising: receiving a set of raw surgical videos corresponding to a surgical procedure performed within the OR; merging the set of raw surgical videos to generate a surgical procedure video corresponding to the surgical procedure, wherein the surgical procedure video comprises a set of raw video images; detecting image-based personally-identifiable information embedded in the set of raw video images, which includes detecting a type of image-based personally-identifiable information associated with one or more surgical tools used during the surgical procedure, wherein the set of raw video images includes images of the one or more surgical tools captured inside a patient's body when used to perform the surgical procedure; and when image-based personally-identifiable information is detected, de-identifying the detected image-based personally-identifiable information in the surgical procedure video.
The computer-implemented method of claim 1, wherein merging the set of raw surgical videos to generate the surgical procedure video includes: analyzing a set of file names associated with the set of raw surgical videos to determine a correct order with respect to the surgical procedure; and stitching together the set of raw surgical videos based on the determined order.
The computer-implemented method of claim 1, wherein detecting the type of image-based personally-identifiable information associated with the one or more surgical tools further includes: detecting personally-identifiable text printed on the one or more surgical tools in the images of the one or more surgical tools; wherein if personally-identifiable text is detected in the images of the one or more surgical tools, de-identifying the detected image-based personally-identifiable information by blurring out or otherwise making the detected text unreadable in the images of the one or more surgical tools.
The computer-implemented method of claim 1, wherein the image-based personally-identifiable information further includes: text displayed inside the OR where the surgical procedure was performed, wherein the text was recorded by the recording device during an out-of-body (OOB) event when the recording device was taken out of the patient's body.
The computer-implemented method of claim 3, wherein detecting the personally-identifiable text printed on the one or more surgical tools includes: detecting, using a machine-learning-based tool-detection-and-recognition model, a surgical tool within a first subset of the set of raw video images; and processing a portion of the first subset of the set of raw video images containing the detected surgical tool to detect any personally-identifiable text within the portion of the first subset of the set of raw video images.
The computer-implemented method of claim 1, wherein the image-based personally-identifiable information further includes a text box inserted into the surgical procedure video to display surgical procedure-related information.
The computer-implemented method of claim 6, wherein detecting the text-box in the surgical procedure video includes: detecting, using a machine-learning-based text-box detection model, a text box at or in the vicinity of a predetermined location within a second subset of the set of raw video images; and processing a portion of the second subset of the set of raw video images containing the detected text box to detect any personally-identifiable text within the portion of the second subset of the set of raw video images.
The computer-implemented method of claim 1, wherein detecting the image-based personally-identifiable information embedded in the set of raw video images of the surgical procedure video further includes: scanning the set of raw video images to detect a video segment corresponding to an OOB event when the recording device was taken out of a patient's body, wherein personally-identifiable information can be captured by the recording device during the OOB event; and wherein if an OOB event is detected within the set of raw video images, de-identifying the detected image-based personally-identifiable information by automatically blurring out or otherwise edited out each video image in the detected video segment corresponding to the detected OOB event, so that any personally-identifiable information embedded within the detected video segment cannot be identified.
The computer-implemented method of claim 8, wherein detecting a video segment corresponding to an OOB event in the set of raw video images includes: detecting, using a machine-learning-based OOB-event detection model, a beginning phase of an OOB event when an endoscope camera is being taken out of a patient's body; detecting, using the machine-learning-based OOB-event detection model, an ending phase of the OOB event when the endoscope camera is being inserted back into the patient's body; and marking a set of video images between the detected beginning phase and the detected ending phase in the set of raw video images as the video segment corresponding to the detected OOB event.
The computer-implemented method of claim 9, wherein prior to using the machine-learning-based OOB-event detection model to detect an OOB event, the method further comprises training the OOB-event detection model based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.
The computer-implemented method of claim 1, wherein prior to merging the set of raw surgical videos, the method further includes: processing the received set of raw surgical videos to detect text-based personally-identifiable information embedded in file-structure data associated with the set of raw surgical videos; and if text-based personally-identifiable information is detected, removing or otherwise de-identifying the detected text-based personally-identifiable information from the file-structure data.
The computer-implemented method of claim 11, wherein the file-structure data include: file identifiers associated with the set of raw surgical videos; folder identifiers of a file folder containing the set of raw surgical videos; file properties associated with the set of raw surgical videos; and other metadata associated with the set of raw surgical videos.
The computer-implemented method of claim 1, wherein after de-identifying the detected image-based personally-identifiable information in the surgical procedure video, the method further includes: performing a random sampling within the de-identified surgical procedure video to randomly select a number of video segments within the de-identified surgical procedure video; and verifying that the randomly selected video segments are free of any personally-identifiable information.
The computer-implemented method of claim 1, wherein after de-identifying the detected image-based personally-identifiable information in the surgical procedure video, the method further includes: permanently removing the set of raw surgical videos from a surgical video repository; and replacing the removed raw surgical videos with de-identified surgical procedure video.
The computer-implemented method of claim 1, wherein the personally-identifiable information includes both patient identifiable information associate with a patient receiving the surgical procedure and surgical staff identifiable information associated with surgical staff performing the surgical procedure.
A system for anonymizing raw surgical procedure videos recorded by a recording device within an operating room (OR), the system comprising: one or more processors; a memory coupled to the one or more processors; and wherein the memory stores a set of instructions that, when executed by the one or more processors, cause the system to: receive a set of raw surgical videos corresponding to a surgical procedure performed within the OR; merge the set of raw surgical videos to generate a surgical procedure video corresponding to the surgical procedure, wherein the surgical procedure video comprises a set of raw video images; detect personally-identifiable information embedded in the set of raw video images of the surgical procedure video, which includes detecting a type of image-based personally-identifiable information associated with one or more surgical tools used during the surgical procedure, wherein the set of raw video images includes images of the one or more surgical tools captured inside a patient's body when used to perform the surgical procedure; and when personally-identifiable information is detected, de-identify the detected personally-identifiable information in the surgical procedure video.
The system of claim 16, wherein detecting the type of personally-identifiable information associated with the one or more surgical tools further includes: detecting personally-identifiable text printed on the one or more surgical tools in the images of the one or more surgical tools; and wherein if personally-identifiable text is detected in the images of the one or more surgical tools, de-identifying the detected personally-identifiable information includes blurring out or otherwise making the detected text unreadable in the images of the one or more surgical tools.
The system of claim 16, wherein
[Truncated to input limit]