Testing the Combination of New Anti-cancer Drug Peposertib With Avelumab and Radiation Therapy for Advanced/Metastatic Solid Tumors and Hepatobiliary Malignancies

This phase I/II trial studies the best dose and side effects of peposertib and to see how well it works with avelumab and hypofractionated radiation therapy in treating patients with solid tumors and hepatobiliary malignancies that have spread to other places in the body (advanced/metastatic). Peposertib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Immunotherapy with monoclonal antibodies, such as avelumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. Hypofractionated radiation therapy delivers higher doses of radiation therapy over a shorter period of time and may kill more tumor cells and have fewer side effects. Giving peposertib in combination with avelumab and hypofractionated radiation therapy may work better than other standard chemotherapy, hormonal, targeted, or immunotherapy medicines available in treating patients with solid tumors and hepatobiliary malignancies.

PRIMARY OBJECTIVES:

I. To determine the safety and tolerability and recommended phase 2 dose (RP2D) of peposertib (M3814) in combination with hypofractionated radiation and avelumab in patients with advanced/metastatic solid tumors. (Phase I) II. To determine the efficacy of the combination of hypofractionated radiation, peposertib (M3814), and avelumab as compared to the combination of hypofractionated radiation and avelumab in patients with advanced/metastatic hepatobiliary tumors by objective response rate (ORR) in non-irradiated lesions. (Phase II)

SECONDARY OBJECTIVES:

I. To observe and record anti-tumor activity. (Phase I) II. To characterize the pharmacokinetic (PK) profile of peposertib (M3814) in combination with avelumab. (Phase I) III. To determine the efficacy and safety of the combination of hypofractionated radiation, peposertib (M3814), and avelumab as compared to hypofractionated radiation and avelumab by measurement of disease control rate (DCR), duration of response (DOR), progression free survival (PFS), PFS outside the irradiated field, and overall survival (OS) in patients with advanced/metastatic hepatobiliary tumors. (Phase II) IV. To determine if baseline deoxyribonucleic acid (DNA) repair defects inherent to some cholangiocarcinomas correlate with a more dramatic response to radiation compared to those without as measured by gamma H2AX, phosphorylated (p)NBS1 and pKAP1 immunofluorescence (IFA) with beta CATN segmentation assay. (Phase II) V. To characterize the pharmacokinetic (PK) profiles of peposertib (M3814) and avelumab. (Phase II)

EXPLORATORY OBJECTIVES:

I. To perform molecular profiling assays on malignant and normal tissues, including, but not limited to, whole exome sequencing (WES), ribonucleic acid (RNA) sequencing (RNAseq), mass cytometry (CyTOF), multiplexed ion beam imaging (MIBI), and T cell receptor sequencing in order to:

Ia. To determine if baseline tumor mutation burden and pattern, and neoantigen burden correlate with response.

Ib. To determine if combination therapy results in changes in the immune landscape in both the tumor and the host that correlate with response.

Ic. To determine if baseline defects in deoxyribonucleic acid (DNA) damage repair in some cholangiocarcinomas correlate with an increased response.

OUTLINE: This is a phase I, dose-escalation study of peposertib followed by a phase II study.

PHASE I:

Patients with advanced/metastatic malignant solid tumors undergo 8 fractions of hypofractionated radiation therapy (RT) every day (QD) on days -17 to -7. Patients also receive peposertib orally (PO) twice daily (BID) on days 1-28, and avelumab intravenously (IV) over 60 minutes on days 1 and 15. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity. Patients also undergo computed tomography (CT), biopsy, and collection of blood samples during screening and on study.

PHASE II: Patients advanced/metastatic cholangiocarcinoma/gallbladder cancer are randomized to 1 of 2 arms.

ARM A: Patients undergo 8 fractions of hypofractionated RT QD on days -17 to -7. Patients also receive avelumab IV over 60 minutes on days 1 and 15. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity. Patients also undergo CT, biopsy, and collection of blood samples during screening and on study.

ARM B: Patients undergo 8 fractions of hypofractionated RT QD on days -17 to -7. Patients also receive peposertib PO BID on days 1-28, and avelumab IV over 60 minutes on days 1 and 15. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity. Patients also undergo C), biopsy, and collection of blood samples during screening and on study

After completion of study treatment, patients are followed up at 30 days, every 3 months for 1 year, every 6 months for year 2, then annually thereafter.

Eligibility Criteria:
Inclusion Criteria:

* PHASE 1: Patients must have a histologically confirmed metastatic or locally advanced unresectable solid tumor that has progressed on or after available standard of care therapy or for which no acceptable standard of care therapy exists, or in which the patient declines standard of care therapy (each patient that declines standard of care therapy will be documented in the case report form)
* PHASE 2: Patients must have a histologically confirmed metastatic or locally advanced unresectable cholangiocarcinoma/gallbladder carcinoma that has progressed on gemcitabine, cisplatin, and durvalumab/pembrolizumab.
* Age \>= 18 years

  * Because no dosing or adverse event data are currently available on the use of peposertib (M3814) in combination with avelumab in patients \< 18 years of age
* Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%)
* Patients with at least 1 index lesion to irradiate for whom palliative radiation treatment is indicated (including but not limited to pain and/or symptom control, prevention of disease -related complications, and preservation of organ function). Lung and liver lesions are preferred, though alternate lesions may be considered after discussion with trial principal investigator (PI). Up to 2 lesions may be considered for irradiation provided at least 1 lesion will receive the study treatment of total of 60 Gy and all prescribed irradiation will be completed within the radiation window
* Patients with at least 1 Response Evaluation Criteria in Solid Tumors (RECIST) measurable lesion (to be unirradiated) (defined as those accurately measured in at least one dimension, with the longest diameter to be recorded for non-nodal lesions and the shortest diameter for nodal lesions). Measurable is defined as at least 10 mm in longest diameter for solid tumors, at least 15 mm in shortest diameter for lymph nodes
* Patients must be willing to undergo fresh biopsies at baseline (as opposed to using archival tissue), in the event their baseline tissue was obtained \> 12 months prior to study consent and/or they are randomized to the gamma H2AX, pNBS1 and pKAP1 IFA with beta CATN segmentation assay
* Absolute neutrophil count (ANC) \>= 1,500/mcL
* Platelet count \>= 100,000/mcL
* Hemoglobin \>= 9.0 g/dL
* Serum creatinine =\< 1.5 x upper limit of normal (ULN) OR calculated serum creatinine clearance (glomerular filtration rate \[GFR\] can be used in place of creatinine or creatinine clearance) \>= 60 mL/min for participants with creatinine levels \> 1.5 x institutional ULN

  * Calculate serum creatinine clearance using the standard Cockcroft-Gault formula
* Serum total bilirubin =\< 1.5 x ULN or direct bilirubin =\< ULN for participants with total bilirubin \> 1.5 x ULN

  * Patients with known Gilbert disease with serum bilirubin level =\< 3 x ULN are eligible
* Aspartate aminotransferase (AST) (serum glutamic-oxaloacetic transaminase \[SGOT\]) and alanine aminotransferase (ALT) (serum glutamate pyruvate transaminase \[SGPT\]) =\< 2.5 x ULN or =\< 5.0 x ULN for patients with hepatobiliary tumors/liver metastases
* Albumin \>= 2.8 g/L
* International normalized ratio (INR) or prothrombin time (PT) or activated partial thromboplastin time (aPTT) =\< 1.5 x ULN

  * This applies only to patients not receiving therapeutic anticoagulation; patients receiving therapeutic anticoagulation should be on a stable dose
* Participants must have the ability to swallow and retain oral medication and not have any clinically significant gastrointestinal abnormalities that might alter absorption
* Female patients of childbearing potential must have a negative urine or serum pregnancy test. If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required. The effects of peposertib (M3814) and avelumab on the developing human fetus are unknown and there is the potential for teratogenic or abortifacient effects. For this reason, women and men of child-bearing potential must agree to use adequate contraception (hormonal or barrier method of birth control; abstinence) prior to study entry, for the duration of study treatment, and for 6 months after completion of peposertib (M3814) and avelumab administration. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with peposertib (M3814) and avelumab, breastfeeding should be discontinued if the mother is treated with peposertib (M3814) and avelumab
* Ability to understand and the willingness to sign a written informed consent document. Participants with impaired decision-making capacity (IDMC) who have a close caregiver or legally authorized representative (LAR) and/or family member available will also be eligible

Exclusion Criteria:

* PHASE I: Patients who have received prior anti-CTLA-4, anti-PD-1, anti-PD-L1 or other immune checkpoint inhibitor therapeutic antibodies or pathway-targeting agents
* PHASE II: Patients who have received prior anti-CTLA-4, anti-PD-1, anti-PD-L1 or other immune checkpoint inhibitor therapeutic antibodies or pathway-targeting agents with the following exceptions:

  * Patients who have only received previous durvalumab (anti-PD-L1) in combination with gemcitabine +/- cisplatin as part of first line therapy (TOPAZ-1 regimen) are eligible
  * Patients who have only received previous pembrolizumab (anti-PD-1) in combination with gemcitabine +/- cisplatin as part of first line therapy (KEYNOTE-966 regimen) are eligible
* Patients who have had chemotherapy, definitive radiation, biological cancer therapy, or investigational agent/device within 21 days of first planned dose of study therapy (within 14 days for palliative radiation). Previously irradiated lesions may be re-irradiated provided there is disease progression in the irradiated lesion and the prescribed radiation dosage can safely be re- administered
* Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> Common Terminology Criteria for Adverse Events \[CTCAE\] grade 1) with the exception of alopecia
* Patients with untreated/uncontrolled central nervous system (CNS)/leptomeningeal disease. Patients with asymptomatic, treated CNS disease are eligible if the treating physician determines that immediate CNS specific treatment is not required and is unlikely to be required during the first cycle of therapy and the following criteria are met:

  * Radiographic demonstration of clinical stability upon the completion of CNS-directed therapy and no evidence of interim progression between the completion of CNS-directed therapy and the screening radiographic study done \>= 4 weeks from completion of radiotherapy and \>= 2 weeks from discontinuation of corticosteroids
  * No stereotactic radiation or whole-brain radiation within 28 days prior to randomization
* Patients with active autoimmune disease requiring systemic corticosteroids greater than the equivalent of prednisone 10 mg daily including but not limited to: systemic lupus erythematosus, rheumatoid arthritis, inflammatory bowel disease, colitis, vascular thrombosis associated with antiphospholipid syndrome, Wegener's granulomatosis, Sjogren's syndrome, Bell's palsy, Guillain-Barre syndrome, multiple sclerosis, autoimmune thyroid disease, vasculitis, or glomerulonephritis, with the following exceptions:

  * Patients with a history of autoimmune hypothyroidism on a stable dose of thyroid replacement hormone are eligible
  * Patients with controlled type 1 diabetes mellitus on a stable insulin regimen are eligible
  * Patients with eczema, psoriasis, lichen simplex chronicus of vitiligo with dermatologic manifestations only who require only low potency topical steroids (e.g., hydrocortisone 2.5%, hydrocortisone butyrate 0.1%, fluocinolone 0.01%, desonide 0.05%, alclometasone dipropionate 0.05%) are eligible
* Patients receiving treatment with systemic immunosuppressive medications (including, but not limited to, prednisone, cyclophosphamide, azathioprine, methotrexate, thalidomide, and anti-tumor necrosis factor \[TNF\] agents) within 6 weeks must discontinue these medications prior to starting peposertib (M3814) and avelumab on day 7, with the exception of:

  * Patients with active autoimmune disease managed with systemic corticosteroids less than the equivalent of prednisone 10 mg daily
  * Patients who have received acute, low dose, systemic immunosuppressant medications (e.g., a one-time dose of dexamethasone for nausea)
  * The use of inhaled corticosteroids and mineralocorticoids (e.g., fludrocortisone) for patients with orthostatic hypotension and adrenocortical insufficiency
* Patients who have undergone prior solid organ or bone marrow transplant with the exception of patients with prior renal transplant for whom dialysis may be employed in the event of graft rejection
* Patients with uncontrolled intercurrent illness (e.g., including but not limited to uncontrolled hypertension \[HTN\] \[systolic blood pressure (BP) \> 150, diastolic BP \> 100\], symptomatic congestive heart failure \[CHF\], unstable angina pectoris, ischemic myocardial infarction \[MI\] within 6 months, cardiac arrhythmia, recent transient ischemic attack \[TIA or cerebrovascular accident (CVA)\]) within 6 months
* Patients with serious active infection (e.g. requiring hospitalization and/or intravenous \[IV\] antibiotics) within 4 weeks prior to starting peposertib (M3814) and avelumab, or signs/symptoms of infection or receiving oral or IV antibiotics for the treatment of active systemic infection within 2 weeks prior to starting peposertib (M3814) and avelumab. Patients receiving prophylactic antibiotics are eligible
* Patients with known chronic hepatitis B virus (HBV) infection must have an undetectable viral load on suppressive therapy if indicated. Patients with known chronic hepatitis C (HCV) infection must have been treated and cured. Patients who are currently on curative treatment are eligible if they have an undetectable HCV viral load
* Patients with known human immunodeficiency virus (HIV) are allowed on study provided they have:

  * A stable regimen of highly active anti-retroviral therapy (HAART)
  * No requirement for concurrent antibiotics or antifungal agents for the prevention of opportunistic infection
  * A CD4 count above 250 cells/mcL
  * An undetectable HIV viral load on standard polymerase chain reaction (PCR)-based testing
* Patients with history of idiopathic pulmonary fibrosis, pneumonitis (including drug induced), organizing pneumonia (e.g., bronchiolitis obliterans, cryptogenic organizing pneumonia), or evidence of active pneumonitis on screening chest computed tomography (CT) scan
* Patients with known concurrent malignancy that is expected to require active treatment within two years, or may interfere with the interpretation of the efficacy and safety outcomes of this study in the opinion of the treating investigator. Superficial bladder cancer, nonmelanoma skin cancers, and low-grade prostate cancer not requiring cytotoxic therapy should not exclude participation in this trial. Patients with chronic lymphocytic leukemia (CLL) may be enrolled if they do not require active chemotherapy and their hematologic, renal and hepatic function meets criteria previously mentioned
* Patients with psychiatric illness/social situations that would limit compliance with study requirements
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to peposertib (M3814) or avelumab
* Patients unable to discontinue medications or substances that are potent inhibitors, inducers or sensitive substrates of CYP3A4/5 or CYP2C19 prior to starting peposertib (M3814) and avelumab are ineligible. Medications or substances that are strong inhibitors of CYP3A4/5 or CYP2C19 must be discontinued at least 1 week prior to first peposertib (M3814) dose. Strong inducers of CYP3A4/5 or CYP2C19 must be stopped at least 3 weeks prior to the first dose. Drugs mainly metabolized by CYP3A with a narrow therapeutic index as judged by the investigator must stop at least 1 day prior to first peposertib (M3814) dose. Because the lists of these agents are constantly changing, it is important to regularly consult a frequently updated medical reference. As part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the-counter medicine or herbal product. The primary elimination mechanism of avelumab is proteolytic degradation, thus there are no contraindicated medications with respect to avelumab
* Patients who cannot discontinue concomitant proton-pump inhibitors (PPIs) prior to starting peposertib (M3814) and avelumab. These must be discontinued \>= 5 days prior to starting peposertib (M3814) and avelumab. Patients do not need to discontinue calcium carbonate. H2 blockers are allowed provided they are taken at least 2 hours after peposertib (M3814) dose
* Patients receiving sorivudine or any chemically related analogues (such as brivudine) and not able to discontinue prior to starting peposertib (M3814) and avelumab are excluded
* Pregnant and lactating women are excluded from this study because peposertib (M3814) and avelumab are agents with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with peposertib (M3814) and avelumab, breastfeeding should be discontinued if the mother is treated with peposertib (M3814) and avelumab
* Patients who have received live vaccination within 30 days before starting peposertib (M3814) and avelumab