Testing the Addition of Immunotherapy With Hu5F9-G4 (Magrolimab) to the Usual PARP Inhibitor, Olaparib for Treatment of Metastatic or Recurrent Breast or Castrate-Resistant Prostate Cancer With BRCA Mutations

This phase I/Ib trial studies the side effects and best dose of Hu5F9-G4 (magrolimab) when given in combination with olaparib for the treatment of patients with breast or castrate-resistant prostate cancer that have spread from where they first started (primary site) to other places in the body (metastatic) or have come back after a period of improvement (recurrent) and have mutations in the BRCA1/2 genes. Magrolimab is a monoclonal antibody with potential anticancer activity and the cability to stimulate the immune system and may interfere with the ability of tumor cells to grow and spread. A monoclonal antibody is a type of protein that can bind to certain targets in the body, such as molecules that cause the body to make an immune response (antigens). Olaparib is an inhibitor of PARP, an enzyme that helps repair deoxyribonucleic acid (DNA) when it becomes damaged. Blocking PARP may help keep tumor cells from repairing their damaged DNA, causing them to die. PARP inhibitors are a type of targeted therapy. Combination therapy with magrolimab and olaparib may be safe and effective in treating BRCA-mutated metastatic or recurrent breast or castrate-resistant prostate cancer.

PRIMARY OBJECTIVES:

I. To evaluate the safety, tolerability, toxicity profile (dose-limiting toxicities \[DLTs\], maximum tolerated doses \[MTDs\]), and phase 2 recommended dose (P2RD) of Hu5F9-G4 (magrolimab) combined with olaparib.

II. To evaluate the alteration in the immune microenvironment after combination therapy including alteration of specific immune signatures and pathways related to innate immune response and STING pathway. (Dose expansion)

SECONDARY OBJECTIVES:

I. To evaluate the status of homologous recombination repair (HR)/DNA damage response (DDR) and other genomic mutations.

II. To characterize the pharmacokinetic (PK) profile of olaparib and magrolimab in combination.

EXPLORATORY OBJECTIVES:

I. To explore the activity of the combination regimen in entire cohort and in the dose expansion cohorts in term of objective response rate (ORR), and progression-free survival (PFS).

II. To evaluate the peripheral immune profile. III. To evaluate alterations in the HR/DDR pathway by circulating tumor (ct)DNA.

IV. To correlate drug exposure with response and/or toxicity.

OUTLINE: This is a dose-escalation study of magrolimab in combination with fixed-dose Olaparib followed by a dose-expansion study.

Patients receive magrolimab intravenously (IV) on days 1, 8, 15 and 22 of cycle 1 and days 1 and 15 of subsequent cycles. Patients also receive olaparib orally (PO) twice a day (BID) during each cycle. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity. Patients also undergo collection of blood samples and computed tomography (CT) and/or magnetic resonance imaging (MRI) throughout the trial. Patients in the dose-expansion portion of the study also undergo tumor biopsies during screening and on study.

Patients are followed for 30 days after removal from study treatment, and then followed every 12 months for 36 months or until death whichever comes first.

Eligibility Criteria:
Inclusion Criteria:

* Presence of metastatic and/or recurrent solid tumors (therapy naïve or those with prior therapy) with pathogenic BRCA 1/2 mutated cancers where olaparib is indicated as standard of care therapeutic option (not maintenance) as below:

  * Metastatic breast cancer:

    * Germline - Yes (Y); Somatic - No (N): Must have had prior chemotherapy (chemo) (adjuvant \[adj\], treatment); or if hormone receptor positive (HR+), must have had prior endocrine therapy or deemed inappropriate for endocrine treatment
  * Metastatic castrate-resistant prostate cancer (CPRC)

    * Germline - Y; Somatic - Y: Treatment after progressive disease on anti-androgens
* BRCA mutation status must be confirmed in a Clinical Laboratory Improvement Act (CLIA)-certified laboratory (lab)
* Patients \>= 18 years of age. Because no dosing or adverse event data are currently available on the use of Hu5F9-G4 (magrolimab) in combination with olaparib in patients \< 18 years of age, children are excluded from this study
* Patients may not have had prior PARP inhibitor in the metastatic setting when given for therapeutic purposes. Patients with breast cancer who received adjuvant PARP inhibitor (i) are eligible. Patients who had prior maintenance therapy are eligible.
* At least 4 weeks or 4 prior drug half-lives (whichever is shorter) must have elapsed since completion of the previous systemic therapy
* Expansion Cohort: Willingness and feasibility to undergo pre and post treatment biopsy
* Eastern Cooperative Oncology Group (ECOG) performance status =\< 2 (Karnofsky \>= 60%)
* Absolute neutrophil count \>= 1,500/mcL
* Platelets \>= 100,000/mcL
* Total bilirubin =\< 1.5 x institutional upper limit of normal (ULN)
* Aspartate aminotransferase (AST)(serum glutamic oxaloacetic transaminase \[SGOT\])/alanine aminotransferase (ALT)(serum glutamate pyruvate transaminase \[SGPT\]) =\< 3 x institutional ULN
* Glomerular filtration rate (GFR) \>= 50 mL/min/1.73 m\^2 unless data exists supporting safe use at lower kidney function values, no lower than 30 mL/min/1.73 m\^2
* Hemoglobin (Hgb) \>= 9 g/dL
* Patients infected with human immunodeficiency virus (HIV) who are on effective anti-retroviral therapy with undetectable viral load within 6 months may be eligible for this trial
* For patients with evidence of chronic hepatitis B virus (HBV) infection, the HBV viral load must be undetectable on suppressive therapy, if indicated
* Patients with a history of hepatitis C virus (HCV) infection must have been treated and cured. For patients with HCV infection who are currently on treatment, they are eligible if they have an undetectable HCV viral load
* Patients with treated brain metastases are eligible if follow-up brain imaging after central nervous system (CNS)-directed therapy shows no evidence of progression. Patients with prior brain metastasis that was treated with evidence of resolution or stable disease for 6 months are eligible. Patients are required to be stable and fully tapered off steroids for at least \>= 1 month
* Patients with a prior or concurrent malignancy whose natural history or treatment does not have the potential to interfere with the safety or efficacy assessment of the investigational regimen are eligible for this trial
* Patients with known history or current symptoms of cardiac disease, or history of treatment with cardiotoxic agents, should have a clinical risk assessment of cardiac function using the New York Heart Association Functional Classification. To be eligible for this trial, patients should be class 2B or better
* Women of child-bearing potential, and men who are sexually active and their partner who may become pregnant, must use contraception during treatment and at least for 6 months (women) and 4 months (men) after the last dose of magrolimab and olaparib. Breastfeeding is not allowed during treatment and for one month after receiving the last dose of therapy
* Ability to understand and the willingness to sign a written informed consent document. Legally authorized representatives may sign and give informed consent on behalf of study participants.

Exclusion Criteria:

* Previous anti-CD47 therapy
* Concomitant use of strong CYP3A4 inhibitors/inducers can cause clinically significant drug-interactions; thus, study patients who require the use of these CYP enzymes continuously should be excluded. Study patients need to come off 3 eliminated half-lives of moderate CYP3A4 inhibitors and 5 eliminated half-lives of strong CYP3A4 inhibitors
* Patients who require immunosuppressive treatments for comorbidities are not eligible
* Gastrointestinal pathology or history that adversely impacts the ability to take or absorb oral medication
* Inability to comply with the protocol and/or not willing or who will not be available for follow-up assessments
* Patients who are receiving any other investigational agents
* History of allergic reactions attributed to compounds of similar chemical or biologic composition to magrolimab or other agents used in study
* Patients with uncontrolled intercurrent illness
* Pregnant women are excluded from this study because Hu5F9-G4 (magrolimab) is an anti-CD47 monoclonal antibody agent with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with Hu5F9-G4 (magrolimab), breastfeeding should be discontinued if the mother is treated with Hu5F9-G4 (magrolimab). These potential risks may also apply to other agents used in this study