Office Action 1: Non-Final Rejection (2024-03-13)
Rejection types: 103 - Obviousness, Double Patenting

CLAIMS:
CLAIMS What is claimed is:
1\. A method comprising: receiving surgical video captured by an endoscope that is inserted inside a patient’s body during a surgical procedure performed within an operating room (OR); detecting an out-of-body (OOB) event in a video segment of the surgical video, wherein the OOB event occurs when the endoscope was taken out of a patient’s body while the endoscope is still on; and responsive to detecting the OOB event, de-identifying personally-identifiable information (PII) embedded within the video segment.
2\. The method of claim 1 further comprising detecting the PII based on an analysis of the video segment, wherein the PII is de-identified responsive to detecting the PII.
3\. The method of claim 1, wherein the PII comprises textual information or faces associated with a patient or a surgical staff of the surgical procedure, wherein de-identifying the PII comprises blurring or editing out the textual information or the faces from the video segment.
4\. The method of claim 1, wherein the surgical video comprises a plurality of video frames and the video segment comprises a sequence of one or more video frames of the plurality of video frames, wherein de-identifying the PII comprises blacking out each video frame of the sequence of one or more video frames.
5\. The method of claim 1, wherein detecting the OOB event in the video segment of the surgical video comprises detecting, using an image processing algorithm, 1) a beginning phase of the OOB event within the surgical video when the endoscope is being taken out of the patient’s body and 2) an ending phase of the OOB event within the surgical video when the endoscope is being inserted back into the patient’s body.
6\. The method of claim 5, wherein the image processing algorithm is a machine-learning (ML) based OOB-event detection model that is trained to detect OOB events based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.
7\. The method of claim 5, wherein detecting the OOB event in the video segment of the surgical video further comprises: determining a time interval between the beginning phase and the ending phase within the surgical video; and responsive to determining that the time interval is less than a threshold, determining that the video segment that includes the OOB event is between the beginning phase and the ending phase.
8\. An apparatus comprising: at least one processor; and memory having stored therein instructions that when executed by the at least one processor causes the apparatus to: receive surgical video captured by an endoscope that is inserted inside a patient’s body during a surgical procedure performed within an operating room (OR); detect an out-of-body (OOB) event in a video segment of the surgical video, wherein the OOB event occurs when the endoscope was taken out of a patient’s body while the endoscope is still on; and responsive to detecting the OOB event, de-identify personally-identifiable information (PII) embedded within the video segment.
9\. The apparatus of claim 8, wherein the memory has further instructions to detect the PII based on an analysis of the video segment, wherein the PII is de-identified responsive to detecting the PII.
10\. The apparatus of claim 8, wherein the PII comprises textual information or faces associated with a patient or a surgical staff of the surgical procedure, wherein the instructions to de-identify the PII comprises instructions to blue or edit out the textual information or the faces from the video segment.
11\. The apparatus of claim 8, wherein the surgical video comprises a plurality of video frames and the video segment comprises a sequence of one or more video frames of the plurality of video frames, wherein the instructions to de-identify the PII comprises instructions to black out each video frame of the sequence of one or more video frames.
12\. The apparatus of claim 8, wherein the instructions to detect the OOB event in the video segment of the surgical video comprises instructions to detect, using an image processing algorithm, 1) a beginning phase of the OOB event within the surgical video when the endoscope is being taken out of the patient’s body and 2) an ending phase of the OOB event within the surgical video when the endoscope is being inserted back into the patient’s body.
13\. The apparatus of claim 12, wherein the image processing algorithm is a machine-learning (ML) based OOB-event detection model that is trained to detect OOB events based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.
14\. The apparatus of claim 12, wherein the instructions to detect the OOB event in the video segment of the surgical video comprises instructions to: determine a time interval between the beginning phase and the ending phase within the surgical video; and responsive to determining that the time interval is less than a threshold, determine that the video segment that includes the OOB event is between the beginning phase and the ending phase.
15\. A non-transitory machine-readable medium having instructions that when executed by at least one processor causes the at least one processor to: receive surgical video captured by an endoscope that is inserted inside a patient’s body during a surgical procedure performed within an operating room (OR); detect an out-of-body (OOB) event in a video segment of the surgical video, wherein the OOB event occurs when the endoscope was taken out of a patient’s body while the endoscope is still on; and responsive to detecting the OOB event, de-identify personally-identifiable information (PII) embedded within the video segment.
16\. The non-transitory machine-readable medium of claim 15, wherein the PII comprises textual information or faces associated with a patient or a surgical staff of the surgical procedure, wherein the instructions to de-identify the PII comprises instructions to blue or edit out the textual information or the faces from the video segment.
17\. The non-transitory machine-readable medium of claim 15, wherein the surgical video comprises a plurality of video frames and the video segment comprises a sequence of one or more video frames of the plurality of video frames, wherein the instructions to de-identify the PII comprises instructions to black out each video frame of the sequence of one or more video frames.
18\. The non-transitory machine-readable medium of claim 15, wherein the instructions to detect the OOB event in the video segment of the surgical video comprises instructions to detect, using an image processing algorithm, 1) a beginning phase of the OOB event within the surgical video when the endoscope is being taken out of the patient’s body and 2) an ending phase of the OOB event within the surgical video when the endoscope is being inserted back into the patient’s body.
19\. The non-transitory machine-readable medium of claim 18, wherein the image processing algorithm is a machine-learning (ML) based OOB-event detection model that is trained to detect OOB events based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.
20\. The non-transitory machine-readable medium of claim 18, wherein the instructions to detect the OOB event in the video segment of the surgical video comprises instructions to: determine a time interval between the beginning phase and the ending phase within the surgical video; and responsive to determining that the time interval is less than a threshold, determine that the video segment that includes the OOB event is between the beginning phase and the ending phase.

OFFICE ACTION:
DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The present application, filed on May 18, 2023, is accepted.
Claims 1 – 20 are being considered on the merits.
Drawings
The drawings, filed on May 18, 2023, are accepted.
Specification
The specification, filed on May 18, 2023, is accepted.
Double Patenting
Claims 1 – 20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 – 19 of U.S. Patent No. 11080424 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because they both disclose similar technique to anonymize surgical/medical video that has personally-identifiable information due to the endoscope being out of the body/patient.
18/320,127
U.S. Patent No. 11080424 B2
1. A method comprising: receiving surgical video captured by an endoscope that is inserted inside a patient’s body during a surgical procedure performed within an operating room (OR); detecting an out-of-body (OOB) event in a video segment of the surgical video, wherein the OOB event occurs when the endoscope was taken out of a patient’s body while the endoscope is still on; and responsive to detecting the OOB event, de-identifying personally-identifiable information (PII) embedded within the video segment.
1. A computer-implemented method for anonymizing raw surgical procedure videos recorded by a recording device within an operating room (OR), the method comprising: receiving a set of raw surgical videos corresponding to a surgical procedure performed within the OR; merging the set of raw surgical videos to generate a surgical procedure video corresponding to the surgical procedure, wherein the surgical procedure video comprises a set of raw video images; detecting image-based personally-identifiable information embedded in the set of raw video images, which includes detecting a type of image-based personally-identifiable information associated with one or more surgical tools used during the surgical procedure, wherein the set of raw video images includes images of the one or more surgical tools captured inside a patient's body when used to perform the surgical procedure; and when image-based personally-identifiable information is detected, de-identifying the detected image-based personally-identifiable information in the surgical procedure video.
2. The method of claim 1 further comprising detecting the PII based on an analysis of the video segment, wherein the PII is de-identified responsive to detecting the PII.
8. The computer-implemented method of claim 1, wherein detecting the image-based personally-identifiable information embedded in the set of raw video images of the surgical procedure video further includes: scanning the set of raw video images to detect a video segment corresponding to an OOB event when the recording device was taken out of a patient's body, wherein personally-identifiable information can be captured by the recording device during the OOB event; and wherein if an OOB event is detected within the set of raw video images, de-identifying the detected image-based personally-identifiable information by automatically blurring out or otherwise edited out each video image in the detected video segment corresponding to the detected OOB event, so that any personally-identifiable information embedded within the detected video segment cannot be identified.
3. The method of claim 1, wherein the PII comprises textual information or faces associated with a patient or a surgical staff of the surgical procedure, wherein de-identifying the PII comprises blurring or editing out the textual information or the faces from the video segment.
8. The computer-implemented method of claim 1, wherein detecting the image-based personally-identifiable information embedded in the set of raw video images of the surgical procedure video further includes: scanning the set of raw video images to detect a video segment corresponding to an OOB event when the recording device was taken out of a patient's body, wherein personally-identifiable information can be captured by the recording device during the OOB event; and wherein if an OOB event is detected within the set of raw video images, de-identifying the detected image-based personally-identifiable information by automatically blurring out or otherwise edited out each video image in the detected video segment corresponding to the detected OOB event, so that any personally-identifiable information embedded within the detected video segment cannot be identified.
4. The method of claim 1, wherein the surgical video comprises a plurality of video frames and the video segment comprises a sequence of one or more video frames of the plurality of video frames, wherein de-identifying the PII comprises blacking out each video frame of the sequence of one or more video frames.
11. The computer-implemented method of claim 1, wherein prior to merging the set of raw surgical videos, the method further includes: processing the received set of raw surgical videos to detect text-based personally-identifiable information embedded in file-structure data associated with the set of raw surgical videos; and if text-based personally-identifiable information is detected, removing or otherwise de-identifying the detected text-based personally-identifiable information from the file-structure data.
5. The method of claim 1, wherein detecting the OOB event in the video segment of the surgical video comprises detecting, using an image processing algorithm, 1) a beginning phase of the OOB event within the surgical video when the endoscope is being taken out of the patient’s body and 2) an ending phase of the OOB event within the surgical video when the endoscope is being inserted back into the patient’s body.
9. The computer-implemented method of claim 8, wherein detecting a video segment corresponding to an OOB event in the set of raw video images includes: detecting, using a machine-learning-based OOB-event detection model, a beginning phase of an OOB event when an endoscope camera is being taken out of a patient's body; detecting, using the machine-learning-based OOB-event detection model, an ending phase of the OOB event when the endoscope camera is being inserted back into the patient's body; and marking a set of video images between the detected beginning phase and the detected ending phase in the set of raw video images as the video segment corresponding to the detected OOB event.
6. The method of claim 5, wherein the image processing algorithm is a machine-learning (ML) based OOB-event detection model that is trained to detect OOB events based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.
10. The computer-implemented method of claim 9, wherein prior to using the machine-learning-based OOB-event detection model to detect an OOB event, the method further comprises training the OOB-event detection model based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.
7. The method of claim 5, wherein detecting the OOB event in the video segment of the surgical video further comprises: determining a time interval between the beginning phase and the ending phase within the surgical video; and responsive to determining that the time interval is less than a threshold, determining that the video segment that includes the OOB event is between the beginning phase and the ending phase.
9. The computer-implemented method of claim 8, wherein detecting a video segment corresponding to an OOB event in the set of raw video images includes: detecting, using a machine-learning-based OOB-event detection model, a beginning phase of an OOB event when an endoscope camera is being taken out of a patient's body; detecting, using the machine-learning-based OOB-event detection model, an ending phase of the OOB event when the endoscope camera is being inserted back into the patient's body; and marking a set of video images between the detected beginning phase and the detected ending phase in the set of raw video images as the video segment corresponding to the detected OOB event.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA  35 U.S.C. 102 and 103 (or as subject to pre-AIA  35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA  to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 – 20 are rejected under 35 U.S.C. 103 as being unpatentable over US 20180110398 A1 to Schwartz et al., (hereinafter, “Schwartz”) in view of US 20200285771 A1 to Dey et al., (hereinafter, “Dey”).
Regarding claim 1, Schwartz teaches a method comprising: receiving surgical video captured by an endoscope that is inserted inside a patient’s body during a surgical procedure performed within an operating room (OR) [Schwartz, para. 27 discloses A video capture system 104 of the system 100 receives a video image from a video input source 102 that originates from an endoscopic imaging system 101 for obtaining a video image from an endoscope positioned in an internal body cavity of a patient. The system 101 may comprise any of a number of commercially-available endoscopic imaging systems, for example including the PINPOINT endoscopic fluorescence imaging system available from NOVADAQ of Bonita Springs, Fla. Imaging may be directed to any of a variety of endoscopic devices, including but not limited to arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes, uteroscopes and colonoscopes.], but Schwartz does not teach detecting an out-of-body (OOB) event in a video segment of the surgical video, wherein the OOB event occurs when the endoscope was taken out of a patient’s body while the endoscope is still on; and responsive to detecting the OOB event, de-identifying personally-identifiable information (PII) embedded within the video segment.
However, Dey does teach detecting an out-of-body (OOB) event in a video segment of the surgical video, wherein the OOB event occurs when the endoscope was taken out of a patient’s body while the endoscope is still on; [Dey, para. 18 discloses Specific artificial intelligence (AI) models may preferably detect, for example, when the medical images or videos of a surgery, are outside the body and recording personally identifiable information (PII), such as people's faces.] and responsive to detecting the OOB event, de-identifying personally-identifiable information (PII) embedded within the video segment. [Dey, para. 18 discloses the system AI models mark these segments containing PII, remove them from the medical video or image files, and then re-assemble the medical data files, images and video, and outputs the data for the next processing stage.]
Therefore, it would have been obvious to one of ordinary skill within the art before the effective filling date to combine Dey’s system with Schwartz’s system, with a motivation for artificial intelligence (AI) powered software and image recognition algorithms, and models are applied to the medical imaging data to detect personally identifiable information (PII). [Dey, para. 19]
Regarding claim 2, modified Schwartz teaches the method of claim 1, but Schwartz does not teach further comprising detecting the PII based on an analysis of the video segment, wherein the PII is de-identified responsive to detecting the PII.
However, Dey does teach further comprising detecting the PII based on an analysis of the video segment, wherein the PII is de-identified responsive to detecting the PII. [Dey, para. 18 discloses the system preferably uses a multi-pass approach to removed embedded personally identifiable information tags in the medical data files. Specific artificial intelligence (AI) models may preferably detect, for example, when the medical images or videos of a surgery, are outside the body and recording personally identifiable information (PII), such as people's faces.]
Therefore, it would have been obvious to one of ordinary skill within the art before the effective filling date to combine Dey’s system with Schwartz’s system, with a motivation for artificial intelligence (AI) powered software and image recognition algorithms, and models are applied to the medical imaging data to detect personally identifiable information (PII). [Dey, para. 19]
Regarding claim 3, modified Schwartz teaches the method of claim 1, but Schwartz does not teach wherein the PII comprises textual information or faces associated with a patient or a surgical staff of the surgical procedure, wherein de-identifying the PII comprises blurring or editing out the textual information or the faces from the video segment.
However, Dey does teach wherein the PII comprises textual information or faces associated with a patient or a surgical staff of the surgical procedure, wherein de-identifying the PII comprises blurring or editing out the textual information or the faces from the video segment. [Dey, para. 18 discloses the system preferably uses a multi-pass approach to removed embedded personally identifiable information tags in the medical data files. Specific artificial intelligence (AI) models may preferably detect, for example, when the medical images or videos of a surgery, are outside the body and recording personally identifiable information (PII), such as people's faces. Para. 20 discloses all personally identifiable information (PII) is removed, deleted, erased, censored, bleeped, cut, expunged or redacted from the medical imaging data. PII present in medical imaging video may be deleted on a frame by frame basis, which may include video frames, image stills, image segments, as well as audio data. In the instances of medical images, PII may be removed by deleting entire images, or redacting portions of images.]
Therefore, it would have been obvious to one of ordinary skill within the art before the effective filling date to combine Dey’s system with Schwartz’s system, with a motivation for artificial intelligence (AI) powered software and image recognition algorithms, and models are applied to the medical imaging data to detect personally identifiable information (PII). [Dey, para. 19]
Regarding claim 4, modified Schwartz teaches the method of claim 1, but Schwartz does not teach wherein the surgical video comprises a plurality of video frames and the video segment comprises a sequence of one or more video frames of the plurality of video frames, wherein de-identifying the PII comprises blacking out each video frame of the sequence of one or more video frames.
However, Dey does teach wherein the surgical video comprises a plurality of video frames and the video segment comprises a sequence of one or more video frames of the plurality of video frames, [Dey, para. 9 discloses the medical video data may be organized into individual frames and processed on a frame by frame basis for identifying different activity from the frames, such as periods of detectable PII. The video data may be further edited and reconstructed without the frames and periods of detectable PII.] wherein de-identifying the PII comprises blacking out each video frame of the sequence of one or more video frames. [Dey, para. 20 discloses all personally identifiable information (PII) is removed, deleted, erased, censored, bleeped, cut, expunged or redacted from the medical imaging data. PII present in medical imaging video may be deleted on a frame by frame basis, which may include video frames, image stills, image segments, as well as audio data. In the instances of medical images, PII may be removed by deleting entire images, or redacting portions of images.]
Therefore, it would have been obvious to one of ordinary skill within the art before the effective filling date to combine Dey’s system with Schwartz’s system, with a motivation for artificial intelligence (AI) powered software and image recognition algorithms, and models are applied to the medical imaging data to detect personally identifiable information (PII). [Dey, para. 19]
Regarding claim 5, modified Schwartz teaches the method of claim 1, but Schwartz does not teach wherein detecting the OOB event in the video segment of the surgical video comprises detecting, using an image processing algorithm, 1) a beginning phase of the OOB event within the surgical video when the endoscope is being taken out of the patient’s body and 2) an ending phase of the OOB event within the surgical video when the endoscope is being inserted back into the patient’s body.
However, Dey does teach wherein detecting the OOB event in the video segment of the surgical video comprises detecting, using an image processing algorithm, 1) a beginning phase of the OOB event within the surgical video when the endoscope is being taken out of the patient’s body and 2) an ending phase of the OOB event within the surgical video when the endoscope is being inserted back into the patient’s body. [Dey, para. 18 discloses specific artificial intelligence (AI) models may preferably detect, for example, when the medical images or videos of a surgery, are outside the body and recording personally identifiable information (PII), such as people's faces. The system AI models mark these segments containing PII, remove them from the medical video or image files, and then re-assemble the medical data files, images and video, and outputs the data for the next processing stage. Para. 19 discloses the medical imaging data may be deconstructed into individual frames for detecting PII on a frame by frame basis, and preferably the system may label, mark, tag, or code the beginning and ending of segments of video data frames in which PII is present. In a preferred application of the system and method AI models, the medical imaging data will be labeled and coded on a frame-level basis for the presence of PII.]
Therefore, it would have been obvious to one of ordinary skill within the art before the effective filling date to combine Dey’s system with Schwartz’s system, with a motivation for artificial intelligence (AI) powered software and image recognition algorithms, and models are applied to the medical imaging data to detect personally identifiable information (PII). [Dey, para. 19]
Regarding claim 6, modified Schwartz teaches the method of claim 5, but Schwartz does not teach wherein the image processing algorithm is a machine-learning (ML) based OOB-event detection model that is trained to detect OOB events based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos.
However, Dey does teach wherein the image processing algorithm is a machine-learning (ML) based OOB-event detection model that is trained to detect OOB events based on a set of labeled video segments of a set of OOB events extracted from actual surgical procedure videos. [Dey, para. 19 discloses the AI models are applied to find and detect any instances of personally identifiable information, which may preferably include any health care information or medical history, which can be linked to any individual, and additionally, patient names, medical professional names, faces, home addresses, email addresses, telephone numbers, social security numbers, account numbers, digital identities, passwords, login information, screen names, dates of birth, age, genetic information, or biometric data. The medical imaging data may be deconstructed into individual frames for detecting PII on a frame by frame basis, and preferably the system may label, mark, tag, or code the beginning and ending of segments of video data frames in which PII is present. In a preferred application of the system and method AI models, the medical imaging data will be labeled and coded on a frame-level basis for the presence of PII.]
Therefore, it would have been obvious to one of ordinary skill within the art before the effective filling date to combine Dey’s system with Schwartz’s system, with a motivation for artificial intelligence (AI) powered software and image recognition algorithms, and models are applied to the medical imaging data to detect personally identifiable information (PII). [Dey, para. 19]
Regarding claim 7, modified Schwartz teaches the method of claim 5, but Schwartz does not teach wherein detecting the OOB event in the video segment of the surgical video further comprises: determining a time interval between the beginning phase and the ending phase within the surgical video; and responsive to determining that the time interval is less than a threshold, determining that the video segment that includes the OOB event is between the beginning phase and the ending phase.
However, Dey does teach wherein detecting the OOB event in the video segment of the surgical video further comprises: determining a time interval between the beginning phase and the ending phase within the surgical video; [Dey, para. 18 discloses specific artificial intelligence (AI) models may preferably detect, for example, when the medical images or videos of a surgery, are outside the body and recording personally identifiable information (PII), such as people's faces. The system AI models mark these segments containing PII, remove them from the medical video or image files, and then re-assemble the medical data files, images and video, and outputs the data for the next processing stage.] and responsive to determining that the time interval is less than a threshold, determining that the video segment that includes the OOB event is between the beginning phase and the ending phase. [Dey, Para. 19 discloses the medical imaging data may be deconstructed into individual frames for detecting PII on a frame by frame basis, and preferably the system may label, mark, tag, or code the beginning and ending of segments of video data frames in which PII is present. In a preferred application of the system and method AI models, the medical imaging data will be labeled and coded on a frame-level basis for the presence of PII.]
Therefore, it would have been obvious to one of ordinary skill within the art before the effective filling date to combine Dey’s system with Schwartz’s system, with a motivation for artificial intelligence (AI) powered software and image recognition algorithms, and models are applied to the medical imaging data to detect personally identifiable information (PII). [Dey, para. 19]
Regarding claims 8 – 14, they recite features similar to features within claims 1 – 7, therefore, they are rejected in similar manner.
Regarding claim 15, it recites features similar to features within claim 1, therefore, it is rejected in similar manner.
Regarding claims 16 – 20, they recite features similar to features within claims 3 – 7, therefore, they are rejected in similar manner.
Conclusion
Pertinent prior art made of record however not relied upon:
US 20200285771 A1 to Dubin et al.
“A system and method for removing personally identifiable information (PII) from medical data and ensuring strict compliance with privacy laws and regulations. The PII removal tool removes, deletes or redacts sensitive personally identifiable information, which may comprise any health care data that may be linked to any individual. The system and method is preferably applied to medical data, images, video, or other electronic media captured from medical procedures, endoscopes, or arthroscopic surgery. Artificial intelligence powered software and models are applied to the medical data for the detection of PII comprising patient names, medical history, faces, account numbers, digital identities, biometric data, or other sensitive personal information which may be linked to an individual. Medical data is re-written without PII and the system generates metadata describing the deleted PII. Privacy law compliant medical data is normalized, stored hierarchically, and transmitted to third party storage systems for analysis with software models for medical insights, anatomical recognition, surgical metrics, patient outcome, diagnosis and medical payment and billing.”

APPLICANT REBUTTAL: REMARKS
In response to the above-identified Office Action, Applicant has amended claims 1, 2, and 4-20. Applicant has not canceled or added any new claims. Support for these amendments can be found in the originally filed application. Thus, no new matter has been added. Accordingly, claims 1-20 will be pending in the present application. In view of the remarks presented herein, Applicant requests reconsideration and allowance of the present application.
I.
Applicant's Statement of the Substance of the Interview
Applicant's representative had a telephonic interview with Examiner Pham and Zoubair on June 5, 2024. Applicant's representative Matthew Chavez (Reg. No. 68,318) participated inthe interview. During the interview, Applicant's representative discussed possible claim amendments to the independent claims and proposed remarks in response to the rejections presented in the Office Action. No agreement was reached.
Applicant's representatives did not present any exhibit or conduct any demonstrations. Applicant thanks the Examiners for the telephonic interview. The present Applicant's statement of the substance of the interview is submitted along with the response to the pending Office Action and is therefore timely.
II.
Double Patentin
g
Claims 1-20 stand rejected on the ground of non-statutory obviousness-type double patenting as allegedly being unpatentable over claims 1-19 of U.S. Patent No. 11,080,424. Applicant respectfully requests that the nonstatutory double patenting rejections be held in abeyance until it is determined that the claims are otherwise allowed.
III.
35 U.S.C.
103: Rejection of Claims 1-20
Claims 1-20 are rejected under 35 U.S.C. §103 as being unpatentable over United States Patent Publication 2018/0110398 by Schwartz in view of United States Patent Publication 2020/0285771 by Dey.
A.
Inde
p
endent Claims
Claim 1 recites a method that includes:
receiving surgical video captured by an endoscope that is inserted inside a
patient's body during a surgical procedure performed within an operating room
(OR);
detecting an out-of-body (OOB) event in a video segment of the surgical
video that occurs when the endoscope was taken out of a patient's body while the
endoscope is still on by:
detecting, using an image processing algorithm, 1) afield of view of the endoscope that transitions from within the patient's body to outside the patient's body in a first set of video frames of the surgical video as beginning phase of the OOB event and 2) a subsequent field of view of the endoscope that transitions from outside the patient's body to inside the patient's body in a second set of video
frames of the surgical video as an ending phase of the O0B event, and
determining that a time interval between the beginning phase and the ending phase is within a threshold period of time; and
responsive to detecting the OB event, de-identifying personally- identifiable information (P11) embedded within the video segment.
In other words, the OOB event is detected using an image processing algorithm to detect 1) a beginning phase of the OOB event in a first set of video frames captured by an endoscope and 2) an ending phase of the OOB event in a subsequent set of video frames captured by the endoscope, and determining that a time interval between both phases are within a threshold period of time. Applicant respectfully submits that the relied upon sections of the relied upon references do not disclose or suggest such features.
In rejecting similar features, the Office Action cites paragraphs [0018] and [0019] of Dey. See Office Action, pages 13-16. Applicant respectfully disagrees. The cited portions of Dey describe how a specific Al model detects people's faces that are outside a human body. Nowhere, however, do the cited sections of Dey disclose or suggest detecting an OOB event by 1) using an image processing algorithm to detect beginning and ending phases within different sets of video frames and 2) determining that a time interval between those phases are within a threshold period of time.
In addition, Applicant respectfully submits that the relied upon sections of the relied
upon references do not disclose or suggest detecting the OOB event by determining that a time interval between the beginning phase and the ending phase is within a threshold period of time.
In rejecting similar features, the Office Action appears to assert that paragraph [0019] discloses such features. Applicant respectfully disagrees. The cited portion of Dey describes that medical imaging data may be deconstructed into individual frames for detecting PII on a frame by frame basis, and that the system may label, mark, tag, or code the beginning and ending of segments of video data frames in which PII is present. Thus, Dey is describing tagging segments of video data frames in which PII is present, which is in contrast to the claimed features in which OOB events are detected upon determining that a time interval between the beginning and ending phases is within a threshold period of time.
For at least the foregoing reasons, the cited sections of the relied upon references fail to teach each of the limitations of claim 1.
Applicant respectfully submits that the dependent claims are patentable and non-obvious in light of the relied upon references.
Independent claims 8 and 15 are submitted as not being obvious for at least the same reasons given above in support of independent claim 1.
B.
Dependent Claims
Regarding dependent claim 2, Applicant respectfully submits that the relied upon sections of the relied upon references do not disclose or suggest wherein the video segment comprises a plurality of video frames that make up the O0B event, wherein the PHI that is de- identified is within a portion but not all of the plurality of video frames. Instead, the cited portions of Dey describe identifying PII, such as people's faces, so as to apply additional processing to remove any of the PII. See paragraph [0018]. The identification may include labeling, marking, tagging, or coding video data frames in which the PII is present. See paragraph [0019]. Thus, Dey is only identifying and processing video data frames that include PII, which is in contrast to the claimed features in which a portion of a detected OOB event segment includes PII that is being de-identified.
Any dependent claims not mentioned above are submitted as not being anticipated or obvious, for at least the same reasons given above in support of their base claims, but also for additional reasons not given here in the interest of conciseness.
It should be noted that not all of the assertions made in the Office Action, particularly those with respect to the dependent claims, have been addressed here, in the interest of conciseness. Applicant reserves the right to challenge any of the assertions made in the Office Action by the Examiner, with respect to the relied upon art references and how they would relate to Applicant's claim language, including the right to swear behind or otherwise remove an improper art reference.
In view of the foregoing, it is believed that all claims now pending patentably define the subject invention over the prior art of record and are in condition for allowance and such action is earnestly solicited at the earliest possible date.
CONCLUSION

NEXT OFFICE ACTION: DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to communication filed on June 13, 2024.
Status of claims within the present application:
Claims 1 – 20 are pending.
Claims 1 – 2 and 4 – 20 are amended.
Terminal Disclaimer
The terminal disclaimer filed on August 12, 2024 disclaiming the terminal portion of any patent granted on this application which would extend beyond the expiration date of US 11,080,424 B2 has been reviewed and is accepted.  The terminal disclaimer has been recorded.
Response to Arguments
Regarding Claims 1 – 20 that were rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 – 19 of U.S. Patent No. 11080424 B2, applicant’s argument, see page [8], filed on June 13, 2024, have been fully considered and persuasive, therefore, the rejection is withdrawn.
Claims 1 – 20 that were rejected under 35 U.S.C. 103 as being unpatentable over US 20180110398 A1 to Schwartz et al., (hereinafter, “Schwartz”) in view of US 20200285771 A1 to Dey et al., (hereinafter, “Dey”), applicant’s argument, see page [8 – 11], filed on June 13, 2024, have been fully considered and persuasive, therefore, the rejection is withdrawn.
Allowable Subject Matter
Claims 1 – 20 are allowed. The following is an examiner’s statement of reasons for allowance: the following prior arts were yielded during examination of the claims filed on June 13, 2024 in response to office action mailed on March 13, 2024. They do not explicitly teach “occurs when the endoscope was taken out of a patient's body while the endoscope is still on by:  detecting, using an image processing algorithm, 1) a field of view of the endoscope that transitions from within the patient's body to outside the patient's body in a first set of video frames of the surgical video as beginning phase of the OOB event and 2) a subsequent field of view of the endoscope that transitions from outside the patient's body to inside the patient's body in a second set of video frames of the surgical video as an ending phase of the OOB event, and determining that a time interval between the beginning phase and the ending phase is within a threshold period of time;”, but they are in general realm of applicant’s field or endeavor:
Schwartz et al. [US 20180110398 A1]: This is considered closet prior art to the present application that has methodology and technology for automatically recording a video output signal from an endoscopy device. Video image data is received at a video capture device, and transmitted with associated metadata to a server. Prior to transmission, the video capture device analyzes the video image data to identify a recording start indicator as a function of predetermined threshold conditions. Once the indicator is identified, the device sends recording start timestamp to a backend server. The server indexes and stores the recorded video data as a function of the associated metadata in a database, and also records the recording the recording start timestamp data in the database. The device is further able to identify a recording stop indicator, and prepare recording stop timestamp data for transmission to the server.
Schwartz does disclose a video capture system 104 of the system 100 receives a video image from a video input source 102 that originates from an endoscopic imaging system 101 for obtaining a video image from an endoscope positioned in an internal body cavity of a patient. The system 101 may comprise any of a number of commercially-available endoscopic imaging systems, for example including the PINPOINT endoscopic fluorescence imaging system available from NOVADAQ of Bonita Springs, Fla. Imaging may be directed to any of a variety of endoscopic devices, including but not limited to arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes, uteroscopes and colonoscopes.
Dey et al. [US 20200285771 A1]: This prior art discloses methodology for removing personally identifiable information (PII) from medical data and ensuring strict compliance with privacy laws and regulations. The PII removal tool removes, deletes or redacts sensitive personally identifiable information, which may comprise any health care data that may be linked to any individual. The system and method is preferably applied to medical data, images, video, or other electronic media captured from medical procedures, endoscopes, or arthroscopic surgery. Artificial intelligence powered software and models are applied to the medical data for the detection of PII comprising patient names, medical history, faces, account numbers, digital identities, biometric data, or other sensitive personal information which may be linked to an individual. Medical data is re-written without PII and the system generates metadata describing the deleted PII. Privacy law compliant medical data is normalized, stored hierarchically, and transmitted to third party storage systems for analysis with software models for medical insights, anatomical recognition, surgical metrics, patient outcome, diagnosis and medical payment and billing.
Dey does disclose specific artificial intelligence (AI) models may preferably detect, for example, when the medical images or videos of a surgery, are outside the body and recording personally identifiable information (PII), such as people's faces. the system AI models mark these segments containing PII, remove them from the medical video or image files, and then re-assemble the medical data files, images and video, and outputs the data for the next processing stage.
However, none of the prior arts of record independently or in-combination discloses all the limitation of the independent claims 1, 8, and 15 as recited in the amended set of claims being examined.
Therefore, the independent claims are allowable over the prior arts of record. The dependent claims being definite, further limiting, and fully enabled by the specification are also allowed by virtue of their dependence on the independent claims.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be labeled “Comments on Statement of Reasons for Allowance.”
In most cases, the examiner's actions and the applicant's replies make evident the reasons for allowance, satisfying the record as a whole proviso of the rule. This is particularly true when applicant fully complies with 37 CFR 1.111 (b) and (c) and 37 CFR 1.133(b). Thus, where the examiner's actions clearly point out the reasons for rejection and the applicant's reply explicitly presents reasons why claims are patentable over the reference, the reasons for allowance are in all probability evident from the record and no statement should be necessary. Conversely, where the record is not explicit as to reasons, but allowance is in order, then a logical extension of 37 CFR 1.111 and 1.133 would dictate that the examiner should make reasons of record and such reasons should be specific.
Conclusion

CITED REFERENCE SUMMARIES: Ref US20180110398A1: Claims:
A video capture device for automatically recording a video output signal from an endoscopy device during a recording session, comprising: at least one input port configured for receiving the video output signal from the endoscopy device; at least one processor in communication with the at least one input port, a local storage device accessible by the first processor, and at least one output device in communication with the processor and configured for accessing a network, wherein the at least one processor is operable to analyze video image data of the video output signal, identify a recording start indicator for the recording session as a function of predetermined threshold conditions for the video image data, and prepare and transmit recorded video data including the video image data to the network via the at least one output device upon identifying the recording start indicator.
at least one input port configured for receiving the video output signal from the endoscopy device;
at least one processor in communication with the at least one input port,
a local storage device accessible by the first processor, and
at least one output device in communication with the processor and configured for accessing a network,
wherein the at least one processor is operable to analyze video image data of the video output signal, identify a recording start indicator for the recording session as a function of predetermined threshold conditions for the video image data, and prepare and transmit recorded video data including the video image data to the network via the at least one output device upon identifying the recording start indicator.
The video capture device of claim 1 , wherein the at least one processor comprises at least one central processing unit (CPU) and at least one graphics processing unit (GPU).
The video capture device of claim 1 , wherein the endoscopy device is selected from the group consisting of arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes, uteroscopes and colonoscopes.
The video capture device of claim 1 , wherein the at least one processor is further operable to set a recording start timestamp and transmit the recording start timestamp to the network via the at least one output port.
The video capture device of claim 1 , wherein the at least one processor is further operable to identify a recording stop indicator for the recording session as a function of the predetermined threshold conditions, set a recording stop timestamp and transmit the recording stop timestamp to the network via the at least one output port.
The video capture device of claim 1 , wherein the recorded video data further includes metadata associated with the video image data.
The video capture device of claim 6 , wherein the associated metadata includes one or more elements selected from the group consisting of a calendar date, a timestamp, a location and a user identifier.
The video capture device of claim 7 , wherein the user identifier comprises at least one of a physician identifier or a patient identifier.
The video capture device of claim 5 , wherein: the endoscopy device is selected from the group consisting of arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes. uteroscopes and colonoscopes; and the threshold conditions for the video image data indicate whether the medical device is displaying video image data internal to a patient.
the endoscopy device is selected from the group consisting of arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes. uteroscopes and colonoscopes; and
the threshold conditions for the video image data indicate whether the medical device is displaying video image data internal to a patient.
The video capture device of claim 9 , wherein the threshold conditions are based on one or more elements selected from the group consisting of pixel values, frame to frame motion metrics, edge detection indicators, audio signal, neural networks, convolutional neural networks, and recurrent neural networks.
The video capture device of claim 1 , wherein the processor is further capable to prepare the recorded video data by applying one or more of compression processing or encryption processing.
The video capture device of claim 1 , further comprising at least one audio input port.
The video capture device of claim 1 , wherein the output device comprises at least one of a physical output port or a wireless transmitter.
A method for automatically recording a video output signal from an endoscopy device during a recording session, comprising the steps of: receiving video image data of the video output signal at a video capture device; transmitting signals corresponding to the video image data and associated metadata to the server via the video capture device; analyzing the video image data by the video capture device to identify a recording start indicator for the recording session as a function of predetermined threshold conditions; setting a recording start timestamp by the video capture device; transmitting the recording start timestamp data over the network to the server via the video capture device; indexing the recorded video data by the server as a function of the associated metadata; storing the indexed video data for the recording session in a database accessible to the server; and storing the recording start timestamp for the recording session in the database.
receiving video image data of the video output signal at a video capture device;
transmitting signals corresponding to the video image data and associated metadata to the server via the video capture device;
analyzing the video image data by the video capture device to identify a recording start indicator for the recording session as a function of predetermined threshold conditions;
setting a recording start timestamp by the video capture device;
transmitting the recording start timestamp data over the network to the server via the video capture device;
indexing the recorded video data by the server as a function of the associated metadata;
storing the indexed video data for the recording session in a database accessible to the server; and
storing the recording start timestamp for the recording session in the database.
The method of claim 14 , further comprising the steps of: receiving audio data from at least one microphone; and transmitting signals corresponding to the audio data, associated metadata and an associated timestamp to the server via the video capture device.
receiving audio data from at least one microphone; and
transmitting signals corresponding to the audio data, associated metadata and an associated timestamp to the server via the video capture device.
The method of claim 14 , wherein the step of transmitting signals corresponding to the video image data and associated metadata begins substantially coincidentally with the step of transmitting recording start timestamp data.
The method of claim 14 , wherein the step of transmitting signals corresponding to the video image data and associated metadata occurs continuously.
The method of claim 14 , wherein the endoscopy device is selected from the group consisting of arthroscopes, cystoscopes, hysteroscopes, laparoscopes, laryngoscopes, endoscopes, resectoscopes, sinuscopes, uteroscopes and colonoscopes.
The method of claim 14 , further comprising the steps of: analyzing the video image data by the video capture device to identify a recording stop indicator as a function of the predetermined threshold conditions; setting a recording stop timestamp for the recording session by the video capture device; transmitting recording stop timestamp data to the server over the network via an output port of the video capture device; storing the recording start timestamp by the server in the database.
analyzing the video image data by the video capture device to identify a recording stop indicator as a function of the predetermined threshold conditions;
setting a recording stop timestamp for the recording session by the video capture device;
transmitting recording stop timestamp data to the server over the network via an output port of the video capture device;
storing the recording start timestamp by the server in the database.
The method of claim 19 , wherein the step of transmitting signals corresponding to the video image data and associated metadata terminates coincidentally with the step of transmitting recording stop timestamp data.
The method of claim 14 , wherein the associated metadata includes data indicating one or more elements selected from the group consisting of a calendar date, a timestamp, a location, and a user identifier.
The method of claim 21 , wherein the user identifier comprises at least one of a physician identifier or a patient identifier.
The method of claim 19 , wherein the threshold conditions for the video image data indicate whether the medical device is displaying video image data internal to a patient.
The method of claim 23 , wherein the threshold conditions are based on one or more elements selected from the group consisting of pixel values, frame to frame motion metrics, edge detection indicators, audio signal, neural networks, convolutional neural networks, and recurrent neural networks.
The method of claim 14 , further comprising the steps of: providing a network-accessible user portal by the server; receiving one of a recording start indicator or a recording stop indicator for the recording session via the network-accessible user portal; preparing a timestamp for the indicator received by the server; and storing the timestamp for the server in the database.
providing a network-accessible user portal by the server;
receiving one of a recording start indicator or a recording stop indicator for the recording session via the network-accessible user portal;
preparing a timestamp for the indicator received by the server; and
storing the timestamp for the server in the database.
The method of claim 25 , wherein the recording start indicator or recording stop indicator is received by the server via a user interface of the portal by means of at least one of a keyboard device, a mouse, a foot pedal or a voice-activated device.
The method of claim 26 , wherein a recording start indicator is received by the server, further comprising the step of: applying a contention resolution rule to select only one of the recording start timestamp received from the video capture device or the recording start timestamp for the server, as the recording start timestamp for the recording session.
applying a contention resolution rule to select only one of the recording start timestamp received from the video capture device or the recording start timestamp for the server, as the recording start timestamp for the recording session.
Ref US20200285771A1: Claims:
A system for removing personally identifiable information from medical data comprising: a data ingestion pipeline for receiving medical data from a health care provider; artificial intelligence powered software tools and models for detecting personally identifiable information; a software tool for removing, deleting, or redacting personally identifiable information; a software tool for re-writing the medical data without any personally identifiable information (PII) and generating PII metadata; a software tool for normalizing and storing the medical data in a hierarchical structure; and a software tool for ensuring strict compliance with privacy laws and regulations; wherein, the personally identifiable information (PII) is any medical healthcare information which can be linked to any specific individual; wherein, the medical data is medical images, video or other electronic data from endoscopic cameras, arthroscopic surgeries, or other medical procedures or examinations; and wherein the system and software tools are interfaced with the healthcare provider computer and information technology systems.
a data ingestion pipeline for receiving medical data from a health care provider;
artificial intelligence powered software tools and models for detecting personally identifiable information;
a software tool for removing, deleting, or redacting personally identifiable information;
a software tool for re-writing the medical data without any personally identifiable information (PII) and generating PII metadata;
a software tool for normalizing and storing the medical data in a hierarchical structure; and
a software tool for ensuring strict compliance with privacy laws and regulations; wherein, the personally identifiable information (PII) is any medical healthcare information which can be linked to any specific individual; wherein, the medical data is medical images, video or other electronic data from endoscopic cameras, arthroscopic surgeries, or other medical procedures or examinations; and wherein the system and software tools are interfaced with the healthcare provider computer and information technology systems.
The system of claim 1 for removing personally identifiable information from medical data, wherein the data ingestion pipeline provides a user interface and interactive experience, and wherein the received medical data provides an output identifying segments of personally identifiable information and functionality for user applied labels.
The system of claim 1 for removing personally identifiable information from medical data, wherein the software tools are executed in a HIPAA compliant cloud based computer system.
The system of claim 1 for removing personally identifiable information from medical data, wherein the medical data is used to build a massively scaled data set of privacy law compliant medical data for the improved exchange and analytics of healthcare data across patient populations.
The system of claim 1 for removing personally identifiable information from medical data, wherein the system outputs PII-cleansed medical data specific to a certain type of medical or surgical procedure and is used to develop medical insights for improving patient diagnosis and outcome.
The system of claim 1 for removing personally identifiable information from medical data, wherein the software tools are provided with a user interface for adapting the artificial intelligence powered PII detection models based on the generated PII metadata, to ensure strict compliance with privacy laws and reliable detection and removal of PII.
The system of claim 1 for removing personally identifiable information from medical data, wherein the system provides a user configurable PII detection and labeling interface for training the artificial intelligence powered tools for automated removal of PII at scale.
A system and method for removing personally identifiable information (PII) from medical data, comprising the steps of: ingesting medical data from a healthcare provider onto a computer hardware and software system; applying artificial intelligence powered software and models to detect personally identifiable information; removing the detected personally identifiable information; re-writing the medical data without personally identifiable information and generating metadata to describe the deleted PII; and ensuring compliance with privacy laws and regulations.
ingesting medical data from a healthcare provider onto a computer hardware and software system;
applying artificial intelligence powered software and models to detect personally identifiable information;
removing the detected personally identifiable information;
re-writing the medical data without personally identifiable information and generating metadata to describe the deleted PII; and
ensuring compliance with privacy laws and regulations.
The method of claim 8 for removing personally identifiable information (PII) from medical data, wherein the system provides a user interface for the manipulation of the ingested medical data and an output identifying segments of personally identifiable information and functionality for user applied PII-labels.
The method of claim 8 for removing personally identifiable information (PII) from medical data, wherein the steps for ingesting medical data, detecting and removing PII, and re-writing privacy law compliant medical data are executed in a HIPAA compliant cloud based computer system.
The method of claim 8 for removing personally identifiable information (PII) from medical data, wherein the medical data is outputted and used to build a massively scaled data set of privacy law compliant medical data for the improved exchange and analytics of healthcare data across patient populations.
The method of claim 8 for removing personally identifiable information (PII) from medical data, wherein the system and method outputs PII-cleansed medical data specific to a certain type of medical or surgical procedure and is used to develop medical insights for improving patient diagnosis and outcome.
The method of claim 8 for removing personally identifiable information (PII) from medical data, wherein the method provides for a user interface for adapting the artificial intelligence powered PII detection models based on the generated PII metadata, to ensure strict compliance with privacy laws and reliable detection and removal of PII.
The method of claim 8 for removing personally identifiable information (PII) from medical data, wherein the method provides for a user configurable PII detection and labeling interface for training the artificial intelligence powered software and models for automated removal of PII at scale.
A system and method for detecting and removing personally identifiable information (PII) from a patient population of medical data, comprising: a user-interface for ingesting medical data and applying labels to PII; an artificial intelligence powered software model which is trained by PII-labeled medical data for the further detection and labeling of PII across massive datasets representing a patient population; a software tool for processing and removing PII across massive datasets of PII-labeled medical data; a software tool for re-writing massive datasets of medical data without PII and generating PII metadata to ensure compliance with privacy laws and regulations; and a software tool for providing access and exchange amongst third party data analytics for the development of medical insights in privacy law compliant medical data; wherein the system and method increases the mobility and exchange of medical data across health care providers; and wherein patient diagnosis and outcomes are iteratively improved as the system builds massively scaled labeled datasets of actionable intelligence for healthcare providers.
a user-interface for ingesting medical data and applying labels to PII;
an artificial intelligence powered software model which is trained by PII-labeled medical data for the further detection and labeling of PII across massive datasets representing a patient population;
a software tool for processing and removing PII across massive datasets of PII-labeled medical data;
a software tool for re-writing massive datasets of medical data without PII and generating PII metadata to ensure compliance with privacy laws and regulations; and
a software tool for providing access and exchange amongst third party data analytics for the development of medical insights in privacy law compliant medical data; wherein the system and method increases the mobility and exchange of medical data across health care providers; and wherein patient diagnosis and outcomes are iteratively improved as the system builds massively scaled labeled datasets of actionable intelligence for healthcare providers.
The system and method of claim 15 , for detecting and removing personally identifiable information (PII) from a patient population of medical data, wherein the software tools a
[Truncated to input limit]