A Study Combining the Peposertib (M3814) Pill With Standard Chemotherapy in Patients With Ovarian Cancer With an Expansion in High Grade Serous Ovarian Cancer and Low Grade Serous Ovarian Cancer

This phase I trial studies the side effects and best dose of peposertib when given together with pegylated liposomal doxorubicin hydrochloride in treating patients with high or low grade ovarian cancer that has come back after a period of improvement (recurrent). Peposertib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Drugs used in chemotherapy, such as pegylated liposomal doxorubicin hydrochloride, work in different ways to stop the growth of tumor cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading. Giving peposertib and pegylated liposomal doxorubicin hydrochloride may work better in treating patients with ovarian cancer compared to pegylated liposomal doxorubicin hydrochloride alone.

PRIMARY OBJECTIVE:

I. To determine the safety and tolerability of peposertib (M3814) in combination with pegylated liposomal doxorubicin hydrochloride (PLD) and determine the recommended phase 2 dose (RP2D) of the combination in women with recurrent ovarian cancer.

SECONDARY OBJECTIVES:

I. To observe and record anti-tumor activity. II. To evaluate the pharmacokinetics of peposertib (M3814) when given in combination with PLD.

EXPLORATORY OBJECTIVE:

I. To correlate response to treatment (as defined by response rate and progression free survival) with PLD exposure (in area under the curve \[AUC\]) and PLD associated toxicities in women with recurrent high grade serous and low grade serous ovarian cancer treated in the expansion cohorts.

OUTLINE: This is a dose-escalation study of peposertib followed by a dose-expansion study.

Patients receive peposertib orally (PO) twice daily (BID) on days 1-21, days 1-28, or days 1-7 (depending on dose level) and pegylated liposomal doxorubicin hydrochloride intravenously (IV) on day 1. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity.

Patients also undergo computed tomography (CT) scan or magnetic resonance imaging (MRI) during screening and every 8 weeks throughout the study and after 6 months of study treatment, every 12 weeks. Patients undergo echocardiography (ECHO) during screening and every 6 months. Starting in cycle 13, patients undergo ECHO or multigated acquisition (MUGA) scan every 2 cycles. Additionally, patients undergo blood sample collection throughout the study.

After completion of study treatment, patients are followed up for 30 days.

Eligibility Criteria:
Inclusion Criteria:

* DOSE ESCALATION PHASE: Women with recurrent or persistent epithelial ovarian, fallopian tube or primary peritoneal cancer are eligible. This includes, but is not limited to, the following histologic types: serous adenocarcinoma (grade 1,2, or 3/ high grade or low grade), endometrioid adenocarcinoma, carcinosarcoma, mucinous adenocarcinoma, undifferentiated carcinoma, clear cell adenocarcinoma, mixed epithelial adenocarcinoma, transitional cell carcinoma, or adenocarcinoma not otherwise specified

  * NOTE: Patients who have evidence of DDR deficiency /HRD are eligible if they are at the point in their disease course where they are appropriate candidates for single agent Doxil
* EXPANSION PHASE: The expansion phase will simultaneously accrue to 2 cohorts, low grade serous ovarian cancer (LGSOC) and high grade serous ovarian cancer (HGSOC)

  * Patients accrued to the LGSOC cohort will have recurrent or persistent low grade serous ovarian cancer or grade 1 serous ovarian cancer
  * Patients accrued to the HGSOC cohort will have recurrent or persistent high grade serous ovarian cancer
* Patients must have measurable disease by defined Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 criteria
* Prior therapy:

  * Patients must have received at least one prior line of platinum-based chemotherapy
  * Patients can have received an unlimited number of additional lines of chemotherapy, targeted therapy, biologic therapy, or hormonal therapy
  * Any prior therapy directed at the malignant tumor, including chemotherapy, biologic/targeted therapy, immunotherapy, or hormonal therapy must be discontinued at least 4 weeks, one cycle, or 5 half-lives (whichever is shortest) prior to study treatment initiation
* Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of peposertib (M3814) in combination with pegylated liposomal doxorubicin in patients \< 18 years of age, children are excluded from this study, but will be eligible for future pediatric trials
* Patients with platinum-sensitive ovarian cancer are eligible for only the dose expansion phase if their provider feels that PLD would be an appropriate treatment option for them. Patients with platinum-sensitive ovarian cancer should also be offered any higher priority studies for which they are potentially eligible and/or platinum based chemotherapy or a PARP inhibitor if they are eligible for such therapy
* Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%)
* Patients must have a cardiac ejection fraction \>= the institutional lower limit of normal (LLN)
* Hemoglobin \>= 9 g/dL
* Absolute neutrophil count \>= 1,500/mcL
* Platelets \>= 100,000/mcL
* Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN)
* Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional ULN
* Alkaline phosphatase =\< 2.5 x institutional ULN
* Creatinine clearance \> 30 ml/min
* Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial
* For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated
* Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load
* Patients with treated brain metastases are eligible if follow-up brain imaging after central nervous system (CNS)-directed therapy shows no evidence of progression. The patient must be off steroids and clinically stable
* Female patients of childbearing potential must have a negative urine or serum pregnancy test within 72 hours prior to receiving the first dose of study medication. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required

  * The effects of peposertib (M3814) and liposomal doxorubicin on the developing human fetus are unknown and there is the potential for teratogenic or abortifacient effects. For this reason, women and men of child-bearing potential must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study treatment, and for 6 months after completion of peposertib (M3814) administration. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with peposertib (M3814), breastfeeding should be discontinued if the mother is treated with peposertib (M3814)
* Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial
* Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should have a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, patients should be class 2B or better
* Ability to understand and the willingness to sign a written informed consent document. Participants with impaired decision-making capacity (IDMC) who have a legally-authorized representative (LAR) and/or family member available will also be eligible
* Archival formalin-fixed paraffin-embedded (FFPE) tissue collected within the past 36 months prior to registration must be available for submission for deoxyribonucleic acid (DNA)/ribonucleic acid (RNA) analysis

Exclusion Criteria:

* Patients are excluded from the dose-escalation phase of the study if they are eligible for any available therapies known to confer clinical benefit
* Inability to swallow and/or absorb oral medication (patients with a drainage peg are ineligible)
* Patients may not have received prior anthracyclines (doxorubicin or pegylated liposomal doxorubicin) for treatment of their ovarian cancer
* Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1) with the exception of alopecia, thyroid dysfunction, or neuropathy
* Patients who are receiving any other investigational agents within 28 days prior to start of treatment
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to peposertib (M3814) or pegylated liposomal doxorubicin
* Patients who cannot discontinue concomitant medications or herbal supplements that potentially interact with peposertib (M3814)

  * The following categories of medications and herbal supplements must be discontinued prior to starting study treatment:

    * Strong inducers/inhibitors of CYP3A4/5, CYP2C9, and CYP2C19
    * Substrates of CYP3A4/5, CYP1A2, and CYP2B6 with a narrow therapeutic index
  * Use caution with other substrates of CYP3A4/5, CYP1A2, CYP2B6 and substrates of P-gp, BCRP, OCT1, OATP1B1, and OATP1B3 with a narrow therapeutic index. Close monitoring is advised
  * Because the lists of these agents are constantly changing, it is important to regularly consult a frequently-updated medical reference. As part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the-counter medicine or herbal product. Patient Drug Interactions Handout and Wallet Card) should be provided to patients
* Patients who cannot discontinue concomitant proton-pump inhibitors (PPIs). Patients may confer with the study doctor to determine if such medications can be discontinued. These must be discontinued \>= 5 days prior to study treatment. Patients do not need to discontinue calcium carbonate
* Patients receiving sorivudine or any chemically related analogues (such as brivudine) are excluded
* Patients who have received a live attenuated vaccine within 30 days of dosing with peposertib (M3814)
* Patients with uncontrolled intercurrent illness, including but not limited to ongoing or active infection
* Patients with psychiatric illness/social situations that would limit compliance with study requirements
* Pregnant women are excluded from this study because peposertib (M3814) is DNA-PK inhibitor agent with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with peposertib (M3814), breastfeeding should be discontinued if the mother is treated with peposertib (M3814). These potential risks may also apply to other agents used in this study
* Patients with significant (uncontrolled) cardiac conduction abnormalities are excluded