Ipilimumab, Nivolumab, and Talimogene Laherparepvec Before Surgery in Treating Participants With Localized, Triple-Negative or Estrogen Receptor Positive, HER2 Negative Breast Cancer-deleted

This phase I trial studies talimogene laherparepvec given together with ipilimumab and nivolumab before surgery in patients with triple-negative or estrogen receptor positive, HER2 negative localized breast cancer. Ipilimumab and Nivolumab are immune checkpoint inhibitors that enhance immune response towards cancer cells. Talimogene laherparepvec is a modifies human herpes virus 1 that is an oncolytic virus targeting cancer cells and makes tumor microenvironment more immunogenic to promote immune response against cancer. This study will assess the safety and efficacy of talimogene laherparepvec, ipilimumab, and nivolumab, and provide an insight for further improvement of immunotherapy in breast cancer.

PRIMARY OBJECTIVES:

I. To explore the safety of talimogene laherparepvec in combination with nivolumab and ipilimumab in this population assessed by the frequency and severity of adverse events (AEs) using Common Terminology Criteria for Adverse Events (CTCAE) version (v)4.0 toxicity criteria.

SECONDARY OBJECTIVES:

I. Assess tumor response of talimogene laherparepvec in combination with nivolumab and ipilimumab comparing to no treatment, in subjects with localized estrogen receptor (ER) positive and triple negative infiltrating ductal breast cancer.

II. Descriptively analyze the evidence of tumor necrosis and inflammatory infiltration on histopathological examination and immuno-oncological findings in tumor and peripheral blood of patients treated with talimogene laherparepvec in combination with nivolumab and ipilimumab.

EXPLORATORY OBJECTIVES:

I. Descriptively compare tumor response in patients whose tumors are PD-L1 positive (+) and PD-L1 negative (-) at baseline.

II. Determine the baseline mutational load of patients at the start of treatment and relate it to the response to treatment.

III. Determine if hormone receptor status correlates with tumor responses. IV. Determine the number of tumor infiltrating lymphocytes in patient samples at the start and at the end of treatment and relate them to tumor response.

V. Analyze T-cell receptor (TCR) repertoire in tumor infiltrating lymphocytes and peripheral blood mononuclear cells.

OUTLINE:

Participants receive talimogene laherparepvec intratumorally on days 1, 22, and 36, nivolumab intravenously (IV) over 60 minutes on days 1, 15, 29, and 43, and ipilimumab IV over 90 minutes on days 1 and 43 in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, participants are followed up for 30 days.

Eligibility Criteria:
Inclusion Criteria:

* Written informed consent must be obtained from the subject/legal representative prior to performing any protocol-related procedures, including screening evaluations
* Eastern Cooperative Oncology Group (ECOG) performance status (PS) 0-1
* Localized, palpable, biopsy proven triple negative or ER positive HER2 negative infiltrating ductal breast cancer with size \> 1.5 cm by palpation, excluding breast cancer where neoadjuvant chemotherapy is indicated by current guidelines (i.e. inflammatory subtype, etc.)
* Tumor that is palpable and injectable
* Hemoglobin \>= 9 g/dL
* Absolute neutrophil count \>= 1,500/mm\^3
* Platelet count \>= 100,000/mm\^3
* Aspartate aminotransferase (AST) and alanine aminotransferase (ALT) =\< 2.5 x institutional upper limit of normal (ULN)
* Bilirubin =\< 1.5 x ULN; for subjects with documented/suspected Gilbert?s disease, bilirubin =\< 3 x ULN
* Albumin \>= 2.5 g/dl
* Prothrombin time (PT) / International normalized ratio (INR) and partial thromboplastin time (PTT) =\< 1.5 x institutional upper limit of normal unless the patient is on anticoagulant therapy within 28 days prior to enrollment (if the patient is receiving anticoagulant therapy, PT, and a PTT must be within therapeutic range of intended use of anticoagulants)
* Patients must be willing to submit blood and tissue specimens for translational medicine studies
* Patients must be offered the opportunity to participate in specimen banking for future research
* Women of childbearing potential (WOCBP) must have a negative serum or urine pregnancy test (minimum sensitivity 25 IU/L or equivalent units of human chorionic gonadotropin \[HCG\]) within 24 hours prior to the start of study drug
* Women must not be breastfeeding
* Women of childbearing potential (WOCBP) must be willing to use either two adequate barrier methods or a barrier method plus a hormonal method of contraception to prevent pregnancy, or to abstain from heterosexual activity (complete abstinence) throughout the study, starting with visit 1 through 5 months after the last dose of study therapy. Approved contraceptive methods include, for example, intrauterine device, diaphragm with spermicide, cervical cap with spermicide, male condoms, female condoms with spermicide, or oral contraceptives. Spermicides alone are not an acceptable method of contraception. Should a woman become pregnant or suspect she is pregnant while she or her partner is participating in this study, she should inform her treating physician immediately
* Male patients must agree to use an adequate method of contraception, or to abstain from heterosexual activity (complete abstinence) starting with the first dose of study drug through 7 months after the last dose of study therapy

Exclusion Criteria:

* Contraindications to tumor biopsy and injections (coagulopathy, known history of keloid formation, etc.)
* Women who are pregnant or breastfeeding
* Sexually active subjects and their partners unwilling to use male or female latex condom to avoid potential viral transmission during sexual contact while on treatment and within 30 days after treatment with talimogene laherparepvec
* Inability to give informed consent
* History of malignancies except cured basal cell carcinoma, cutaneous squamous cell carcinoma, melanoma in situ, superficial bladder cancer or carcinoma in situ of the cervix; for other malignancies, must be documented to be free of cancer for \>= 2 years. All other cases can be considered on a case by case basis at the discretion of the principal investigator
* Any condition that might interfere with the subject?s participation in the study, safety, or in the evaluation of the study results
* Concurrent enrollment in another clinical study, unless it is an observational (non-interventional) clinical study or the follow-up period of an interventional study
* Prior exposure to any anti-PD-1 or anti-PD-L1 antibody, or any anti-CTLA 4 antibodies
* Patients must not have received prior treatment with talimogene laherparepvec or other oncolytic virus agents
* Patients must not have received any live vaccine within 30 days prior to registration. Seasonal flu vaccines that do not contain live virus are permitted
* Patients must not have an active infection requiring systemic therapy nor a viral infection requiring intermittent treatment with an antiherpetic drug, other than intermittent topical use
* Patients must not have active herpetic skin lesions or prior complications of herpetic infection (e.g., herpetic keratitis or encephalitis) which requires intermittent or chronic treatment with an anti-herpetic drug other than intermittent topical use
* Any concurrent chemotherapy, immunotherapy, biologic or hormonal therapy for cancer treatment. Concurrent use of hormones for non-cancer-related conditions (e.g., insulin for diabetes and hormone replacement therapy) is acceptable
* Patient must not have evidence of any clinically significant immunosuppression such as the following: primary immunodeficiency state such as severe combined immunodeficiency disease; concurrent opportunistic infection; receiving systemic immunosuppressive therapy within 28 days before enrollment with the exceptions of intranasal, topical, and inhaled corticosteroids or oral corticosteroids at physiologic doses not to exceed 10 mg/day of prednisone or equivalent
* Patients must not have known history human immunodeficiency virus (HIV)
* Clinical or laboratory evidence of an active herpetic infection and in patients who require daily antiviral therapy such as acyclovir
* Active or prior documented autoimmune disease within the past 3 years

  * NOTE: Subjects with active, known or suspected autoimmune disease such as vitiligo, type I diabetes mellitus, residual hypothyroidism due to autoimmune condition only requiring hormone replacement, psoriasis not requiring systemic treatment, or conditions not expected to recur in the absence of an external trigger are permitted to enroll
* Active or prior documented inflammatory bowel disease
* History of (non-infectious) interstitial lung disease (ILD)/pneumonitis that required steroids or has current ILD/pneumonitis or where suspected ILD/pneumonitis cannot be ruled out by imaging at screening
* Exposure to any investigational drug within 7 days prior to screening visit or for which 5 half-lives have not elapsed
* Prisoners or subjects who are involuntarily incarcerated

  * Note: under certain specific circumstances a person who has been imprisoned may be included or permitted to continue as a subject
* Subjects who are compulsorily detained for treatment of either a psychiatric or physical (e.g., infectious disease) illness