Testing Low-Dose Common Chemotherapy (Liposomal Doxorubicin) in Combination With an Anti-Cancer Drug, Peposertib, in Advanced Sarcoma

This phase I trial tests the safety, side effects, and best dose of combination therapy with liposomal doxorubicin and peposertib in treating patients with sarcoma that has spread from where it first started, to other places in the body (metastatic), or cannot be removed by surgery (unresectable) and for which no known cure is available (advanced). Doxorubicin is in a class of medications called anthracyclines. Doxorubicin damages the cell's deoxyribonucleic acid (DNA) and may kill cancer cells. It also blocks a certain enzyme needed for cell division and DNA repair. Liposomal doxorubicin is a form of the anticancer drug doxorubicin that is contained inside very tiny, fat-like particles. Liposomal doxorubicin may have fewer side effects and work better than other forms of the drug. Peposertib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. It may also enhance the activity of chemo- and radiotherapy. There is some pre-clinical evidence in animal models that combining peposertib with liposomal doxorubicin can shrink or stabilize certain types of cancer for longer than either drug alone, but it is not known if this will happen in people. Combination therapy with liposomal doxorubicin and peposertib may be effective in treating patients with advanced sarcoma.

PRIMARY OBJECTIVES:

I. To evaluate the safety and tolerability of peposertib in combination with low-dose pegylated liposomal doxorubicin hydrochloride (liposomal doxorubicin) as evaluated by the dose-limiting toxicity (DLT) rate at each tested dose level. (Dose Escalation) II. To determine the recommended phase 2 dose (RP2D) of liposomal doxorubicin and peposertib combination and determine the maximal tolerated dose (MTD) if identified. (Dose Escalation) III. To obtain a more precise determination of adverse events (e.g. dose limiting toxicities estimate at the selected dose). (Dose Expansion)

SECONDARY OBJECTIVES:

I. To assess the pharmacokinetics of peposertib and liposomal doxorubicin used in combination with each other. (Dose Escalation) II. To estimate the objective response rate (ORR) per Response Evaluation Criteria in Solid Tumors (RECIST) 1.1 in patients with leiomyosarcoma treated with peposertib and low-dose liposomal doxorubicin. (Dose Expansion) III. To estimate the progression free survival (PFS) in patients with leiomyosarcoma treated with peposertib and low-dose liposomal doxorubicin. (Dose Expansion) IV. To assess the pharmacokinetics of peposertib and liposomal doxorubicin used in combination with each other. (Dose Expansion) V. To assess whether DNA damage is exaggerated by the low-dose liposomal doxorubicin in combination with peposertib in patients with homologous recombination (HR)-deficiency. (Dose Expansion)

CORRELATIVE OBJECTIVES:

I. To test the hypothesis that soft tissue sarcomas (STS) with homologous recombination deficiency (HRD) like leiomyosarcomas (LMS) will be more susceptible to DNA-protein kinase inhibitor (PKi) in combination with low-dose liposomal doxorubicin.

II. To test the hypothesis that gammaH2AX and pNBS1 can be used as pharmacodynamic biomarkers of response to DNA-PKi in combination with low-dose liposomal doxorubicin.

III. To test the hypothesis that disease activity correlates with circulating tumor DNA levels in the plasma.

OUTLINE: This is a dose-escalation study of peposertib and liposomal doxorubicin followed by a dose-expansion study.

Patients receive peposertib orally (PO) twice daily (BID) on days 1-28 of each cycle and liposomal doxorubicin intravenously (IV) on day 1 of each cycle. Cycles repeat every 28 days in the absence of disease progression, unacceptable toxicity or withdrawal of consent. Patients undergo blood sample collection, and computed tomography (CT) or magnetic resonance imaging (MRI) throughout the trial. Patients also undergo tissue biopsy during screening and on the trial.

Patients are followed for 30 days after removal from study or until death, whichever occurs first.

Eligibility Criteria:
Inclusion Criteria:

* Patients must have histologically confirmed sarcoma that is metastatic or unresectable and for which there is no known curative treatment
* Dose escalation cohort: Patients must have histologic diagnosis of leiomyosarcoma (LMS) or selected soft tissue sarcomas (myxofibrosarcoma \[MFS\], undifferentiated pleomorphic sarcoma \[UPS\], synovial sarcoma, or dedifferentiated liposarcoma \[DDLPS\]). Pathology review and confirmation of diagnosis will occur at the site enrolling the patient on this study
* Dose expansion cohort: Patients must have histology diagnosis of LMS. Pathology review and confirmation of diagnosis will occur at the site enrolling the patient on this study
* Dose escalation cohort: Patients must have evaluable disease that is amenable to biopsy
* Dose expansion cohort: Patients must have disease which is measurable at study entry according to RECIST 1.1 criteria and amenable to biopsy
* Patients must have been treated with at least 1 prior line of therapy. Prior anthracycline use is permitted as long as the cumulative dose prior to enrollment does not exceed 300 mg/m\^2
* Age \>= 18 years. Because no dosing or adverse event data are currently available on the use of peposertib (M3814) in combination with liposomal doxorubicin in patients \< 18 years of age, children are excluded from this study
* Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%) for both dose escalation and dose expansion
* Absolute neutrophil count \>= 1,500/mcL
* Platelets \>= 100,000/mcL
* Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN)
* Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional ULN
* Hemoglobin \>= 8 g/dL
* Glomerular filtration rate (GFR) \>= 51 mL/min/1.73 m\^2 (per institutional estimate based on creatinine level)
* Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy with undetectable viral load within 6 months are eligible for this trial
* For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated
* Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load
* Patients with treated brain metastases are eligible if follow-up brain imaging after central nervous system (CNS)-directed therapy shows no evidence of progression while off steroid support
* Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial
* Patients with known history of clinically significant cardiac disease, or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents should have a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, patients should be class 2B or better and have an left ventricular ejection fraction (LVEF) above the institutional upper limit of normal if LVEF measurement is available
* Female patients of childbearing potential must have a negative urine or serum pregnancy test within 72 hours prior to receiving the first dose of study medication. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required

  * Female patients of childbearing potential must be willing to use an adequate method of contraception for the course of the study through 3 months after the last dose of peposertib (M3814) and 6 months after the last dose of liposomal doxorubicin
* Male patients of reproductive potential must agree to avoid impregnating a partner while receiving study drug and for 3 months after the last dose of peposertib (M3814) and 6 months after the last dose of liposomal doxorubicin by complying with adequate methods of contraception

  * Note: Abstinence is acceptable if this is the usual lifestyle and preferred contraception for the patient
* Ability to understand and the willingness to sign a written informed consent document

Exclusion Criteria:

* Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1) with the exception of alopecia
* Prior palliative radiotherapy within 14 days of cycle 1 day 1 and prior definitive radiotherapy within 42 days of cycle 1 day 1. Adverse effects of radiation therapy must resolve to baseline prior to cycle 1 day 1
* Patients who are receiving any other investigational agents
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to peposertib (M3814) or other agents used in study
* Patients who cannot discontinue concomitant medications or herbal supplements that are strong inhibitors or strong inducers of cytochrome P450 (CYP) isoenzymes CYP3A4/5, CYP2C9, and CYP2C19. Concomitant use of CYP3A4/5 substrates with a narrow therapeutic index are also excluded. Patients may confer with the study doctor to determine if alternative medications can be used. The following categories of medications and herbal supplements must be discontinued for at least the specified period of time before the patient can be treated:

  * Strong inducers of CYP3A4/5 and CYP2C19: \>= 3 weeks prior to study treatment
  * Strong inhibitors of CYP3A4/5 and CYP2C19: \>= 1 week prior to study treatment
  * Substrates of CYP3A4/5 with a narrow therapeutic index: \>= 1 day prior to study treatment
  * Strong inhibitors of CYP2C9: \>= 1 week prior to study treatment
* Patients who cannot discontinue concomitant proton-pump inhibitors (PPIs). Patients may confer with the study doctor to determine if such medications can be discontinued. These must be discontinued \>= 5 days prior to study treatment. Patients do not need to discontinue calcium carbonate
* LVEF measurement and baseline electrocardiogram (ECG) should be performed as clinically indicated based on cardiac risk assessment of the investigator; patients with known LVEF \< the institutional lower limit of normal (LLN) are excluded
* Patients with uncontrolled intercurrent illness
* Patients who cannot swallow tablets whole
* Patients with new or progressive brain metastases (active brain metastases) or leptomeningeal disease are eligible if the treating physician determines that immediate CNS specific treatment is not required and is unlikely to be required during the first cycle of therapy
* Pregnant women are excluded from this study because peposertib (M3814) is an adenosine triphosphate (ATP)-competitive inhibitor of DNA-protein kinase catalytic subunit (PKcs) with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with peposertib (M3814), breastfeeding should be discontinued if the mother is treated with peposertib (M3814). These potential risks may also apply to other agents used in this study
* Patients may not have received prior treatment with a DNA-protein kinase (PK) inhibitor