Radiation Medication (Radium-223 Dichloride) Versus Radium-223 Dichloride Plus Radiation Enhancing Medication (M3814) Versus Radium-223 Dichloride Plus M3814 Plus Avelumab (a Type of Immunotherapy) for Advanced Prostate Cancer Not Responsive to Hormonal Therapy

This phase I/II trial studies the best dose of M3814 when given together with radium-223 dichloride or with radium-223 dichloride and avelumab and to see how well they work in treating patients with castrate-resistant prostate cancer that had spread to other places in the body (metastatic). M3814 may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Radioactive drugs, such as radium-223 dichloride, may carry radiation directly to tumor cells and not harm normal cells. Immunotherapy with monoclonal antibodies, such as avelumab, may help the body's immune system attack the cancer, and may interfere with the ability of tumor cells to grow and spread. This study is being done to find out the better treatment between radium-223 dichloride alone, radium-223 dichloride in combination with M3814, or radium-223 dichloride in combination with both M3814 and avelumab, to lower the chance of prostate cancer growing or spreading in the bone, and if this approach is better or worse than the usual approach for advanced prostate cancer not responsive to hormonal therapy.

PRIMARY OBJECTIVES:

I. To determine the maximum tolerated dose (MTD) of peposertib (M3814) in combination with radium-223 dichloride or in combination with radium-223 dichloride and avelumab in patients with advanced metastatic castrate-resistant prostate cancer (mCRPC) based on dose limiting toxicities (DLTs) in the doublet or triplet combinations. (Phase 1) II. Radiographic progression free survival (rPFS) will be evaluated based on both skeletal and extraskeletal progression following Prostate Cancer Working Group 3 (PCWG3) methodology. (Phase 2)

SECONDARY OBJECTIVES:

I. To determine the time to the first symptomatic skeletal event \[SSE\]. II. To determine the safety of radium-223 dichloride, M3814, and avelumab combination treatment.

III. To observe and record anti-tumor activity. IV. To evaluate progression free survival (PFS) and overall survival (OS). V. To evaluate symptomatic skeletal events (SSE) per standardized case report form (CRF) distinguishing between pathologic and non-pathogenic fractures.

VI. To explore patient-reported symptomatic adverse events (AE) for tolerability of each treatment arm.

VII. To examine the radium-223 dichloride bio-distribution and absorbed dose in each bone metastatic lesions as well as elsewhere in the body including critical organs using dosimetry.

EXPLORATORY OBJECTIVES:

I. To perform molecular profiling assays on malignant and normal tissues, including, but not limited to, whole exome sequencing (WES), and messenger ribonucleic acid (RNA) sequencing (RNAseq), in order to:

Ia. Identify potential predictive and prognostic biomarkers beyond any genomic alteration by which treatment may be assigned, and Ib. Identify resistance mechanisms using genomic deoxyribonucleic acid (DNA)- and RNA-based assessment platforms.

II. To contribute genetic analysis data from de-identified biospecimens to Genomic Data Commons (GDC), a well annotated cancer molecular and clinical data repository, for current and future research; specimens will be annotated with key clinical data, including presentation, diagnosis, staging, summary treatment, and if possible, outcome.

III. To bank plasma and peripheral immune cells from patients to assess predictive biomarkers of response at the Experimental Therapeutics Clinical Trials Network (ETCTN) biorepository at Nationwide Children's Hospital.

IV. To correlate change in level of total alkaline phosphatase, bone-specific alkaline phosphatase, and serum osteocalcin to rPFS and OS.

OUTLINE: This is a phase I, dose-escalation study of peposertib, followed by a phase II study. Patients are randomized to 1 of 3 arms.

ARM A: Patients receive radium-223 dichloride intravenously (IV) over 1 minute on day 1. Treatment repeats every 28 days for up to 6 cycles in the absence of disease progression or unacceptable toxicity. Patients undergo blood sample collection, bone scan, and computed tomography (CT) or magnetic resonance imaging (MRI) throughout the study.

ARM B: Patients receive radium-223 dichloride as in Arm A and peposertib orally (PO) once daily (QD) or twice daily (BID) on days 3-26. Treatment repeats every 28 days for up to 6 cycles in the absence of disease progression or unacceptable toxicity. Patients undergo blood sample collection, bone scan, and CT or MRI throughout the study.

ARM C: Patients receive radium-223 dichloride IV as in Arm A and peposertib PO QD or BID as in Arm B. Patients also receive avelumab IV over 60 minutes on days 1 and 15 of cycles 2-6. Treatment repeats every 28 days for up to 6 cycles in the absence of disease progression or unacceptable toxicity. Patients undergo blood sample collection, bone scan, and CT or MRI throughout the study.

After completion of study treatment, patients are followed up at 30 days and then every 3 months for up to 2 years.

Eligibility Criteria:
Inclusion Criteria:

* PHASE 1: Eastern Cooperative Oncology Group (ECOG) performance status =\< 1 (Karnofsky \>= 70%)
* PHASE 2: ECOG performance status =\< 2 (Karnofsky \>= 60%)
* Unless a patient has had orchiectomy by surgery, the patient is expected to be on antiandrogen therapy (ADT) for "medical castration". ADT needs to be maintained throughout the study. Testosterone level should be checked, and kept consistently lower than 50 ng/dL, similar to that obtained with bilateral orchiectomy
* Progressive castration-resistant prostate cancer with two or more skeletal metastases identified by 99mTC bone scintigraphy. One or more lymph node metastases allowed, but not mandatory. Lymph node metastases in each individually must measure less than 3 cm in the longest dimension. Visible visceral organ metastases are not allowed. A diagnosis of prostate cancer must have been histologically confirmed at any time point
* Baseline prostatic specific antigen (PSA) level of 1 ng/mL or higher with evidence of progressively increasing PSA values (two consecutive increases over the previous reference value)
* Progression after at least one of the following: abiraterone, enzalutamide, apalutamide, darolutamide, or taxane chemotherapy (docetaxel, cabazitaxel). There is no maximum number of prior therapies. Prior immunotherapies (for example, Sipuleucel-T or pembrolizumab) do not exclude the patient from participation
* Age \>= 18 years. Castrate-resistant prostate cancer (CRPC) affects older adults and is rarely encountered in children and adolescents
* Life expectancy \>= 6 months
* Albumin \> 2.5 mg/dL
* Hemoglobin \> 9 g/dL
* Leukocytes \>= 3,000/mcL
* Absolute neutrophil count \>= 1,500/mcL
* Platelets \>= 100,000/mcL
* Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN) (with the exception of \< 3 mg/dL for patients with Gilbert's disease)
* Aspartate aminotransferase (AST)(serum glutamic-oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional ULN
* Creatinine =\< 1.5 x institutional ULN OR
* Glomerular filtration rate (GFR) \>= 40 mL/min/1.73 m\^2
* Human immunodeficiency virus (HIV)-infected patients on effective anti-retroviral therapy (with no medications prohibited by this protocol \[e.g. drug-drug interactions\]) with undetectable viral load within 6 months are eligible for this trial
* For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy (with no medications prohibited by this protocol \[e.g. drug-drug interactions\]), if indicated
* Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load (with no medications prohibited by this protocol \[e.g. drug-drug interactions\])
* Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial
* Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should have a clinical risk assessment of cardiac function using the New York Heart Association Functional classification. To be eligible for this trial, patients should be class 2B or better
* Concomitant use of physiologic corticosteroids is allowed
* Concomitant use of bisphosphonates is allowed (use of bone health agents is mandatory - either denosumab \[preferred\] or bisphosphonates)
* The effects of radium-223 dichloride, M3814, and avelumab on the developing human fetus are unknown. Men treated or enrolled on this protocol must also agree to use adequate contraception prior to the study, for the duration of study participation, and 6 months after completion of Radium-223 dichloride, M3814, and avelumab administration
* Ability to understand and the willingness to sign a written informed consent document. Participants with impaired decision-making capacity (IDMC) who have a legally-authorized representative (LAR) and/or family member available will also be eligible
* Patients must be able to swallow orally administered medication
* Patients with asymptomatic, treated brain metastases are permitted if there is no evidence of progression for at least 4 weeks after central nervous system (CNS)-directed treatment, as ascertained by clinical examination and brain imaging (magnetic resonance imaging \[MRI\] or CT scan) during the screening period

Exclusion Criteria:

* Active autoimmune conditions or patients on chronic immunosuppression due to underlying autoimmune condition
* Patients who have had chemotherapy or radiotherapy within 4 weeks prior to entering the study
* Patients who have not recovered from adverse events due to prior anti-cancer therapy (i.e., have residual toxicities \> grade 1) with the exception of alopecia
* Patients who are receiving any other investigational agents
* Patients who have had previous hemibody external radiation
* Patients who have had systemic radiotherapy with radioisotopes
* Patients who have imminent/established spinal cord compression, pathological fracture in weight bearing bones or bone lesion with soft tissue component unless treated as appropriate with radiation and/or surgery before starting on trial
* Patients who have a history of allergic reactions attributed to compounds of similar chemical or biologic composition to radium-223 dichloride, M3814, or avelumab
* Patients unable to discontinue medications or substances that are potent inhibitors, inducers or sensitive substrates of CYP3A4/5 or CYP2C19 prior to study treatment are ineligible.

  * Medications or substances that are strong inhibitors of CYP3A4/5 or CYP2C19 must be discontinued at least 1 week prior to first M3814 dose.
  * Medications or substances that are strong inducers of CYP3A4/5 or CYP2C19 must be stopped at least 3 weeks prior to the first M3814 dose.
  * Drugs mainly metabolized by CYP3A with a narrow therapeutic index (as judged by the Investigator or authorized designee) must be discontinued at least 1 day prior to first M3814 dose.
  * Note: Because the lists of these agents are constantly changing, it is important to regularly consult a frequently- updated medical reference. As part of the enrollment/informed consent procedures, the patient will be counseled on the risk of interactions with other agents, and what to do if new medications need to be prescribed or if the patient is considering a new over-the- counter medicine or herbal product.
* Radium-223 dichloride should not be given concurrently with abiraterone plus prednisone/prednisolone
* Patients with uncontrolled intercurrent illness
* Patients with psychiatric illness/social situations that would limit compliance with study requirements
* Patients must not have an active infection requiring systemic treatment
* Patients must not use immunosuppressive medication =\< 7 days of registration, EXCEPT for the following:

  * Intranasal, inhaled, topical steroids, or local steroid injection (e.g., intra-articular injection)
  * Systemic corticosteroids at physiologic doses =\< 10 mg/day of prednisone or equivalent
  * Steroids as premedication for hypersensitivity reactions (e.g., CT scan premedication)
* Patients who cannot discontinue concomitant H2 blockers or proton-pump inhibitors (PPIs). Patients may confer with the study doctor to determine if such medications can be discontinued. These must be discontinued \>= 5 days prior to study treatment. Patients do not need to discontinue calcium carbonate
* Patients receiving sorivudine or any chemically related analogues (such as brivudine) are excluded
* Patients with a known history or present osteonecrosis of the jaw