Venetoclax and Decitabine in Treating Participants With Relapsed/Refractory Acute Myeloid Leukemia or Relapsed High-Risk Myelodysplastic Syndrome

This phase II trial studies how well venetoclax and decitabine work in treating participants with acute myeloid leukemia that has come back or does not respond to treatment, or with high-risk myelodysplastic syndrome that has come back. Drugs used in chemotherapy, such as venetoclax and decitabine, work in different ways to stop the growth of cancer cells, either by killing the cells, by stopping them from dividing, or by stopping them from spreading.

PRIMARY OBJECTIVE:

I. To determine the overall response rate (ORR) of venetoclax in combination with 10-day decitabine in patients with refractory/relapsed acute myeloid leukemia (AML); elderly (\> 60 year old) patients with newly diagnosed AML not eligible for intensive chemotherapy; patients with high-risk myelodysplastic syndrome (MDS) with bone marrow blasts between 10% and 20%, relapsed or refractory to prior hypomethylating agent (HMA) therapy, or chronic myelomonocytic leukemia (CMML) with bone marrow blasts \>= 10% regardless of prior therapy; AML patients with prior history of MDS or CMML who received therapy for the MDS or CMML and progressed to AML, and younger patients with newly diagnosed AML with poor risk complex karyotype and/or TP53 deletions/mutations and patients with blastic plasmacytoid dentritic cell neoplasm (BPDCN).

SECONDARY OBJECTIVES:

I. To determine the duration of response, disease-free survival (DFS), and overall survival (OS) of patients with refractory/relapsed AML treated with this combination.

II. To determine the number of patients who achieve a hematologic improvement (HI) in platelets, hemoglobin, or absolute neutrophil count (ANC) and the number of patients who achieve \> 50% reduction in blasts on therapy with venetoclax/10-day decitabine.

III. To determine the safety of venetoclax in combination with 10-day decitabine in patients with refractory/relapsed AML.

IV. To determine the number of patients who transition towards stem cell transplantation upon achieving response with the combination venetoclax/10-day decitabine regimen.

V. To determine the incidence of infectious complications per cycle with venetoclax in combination with 10-day decitabine.

EXPLORATORY OBJECTIVES:

I. To investigate possible relationships between baseline protein and gene expression signatures/mutation profile and BH3 profiling in predicting clinical response to the combination.

II. To characterize the pharmacokinetic (PK) profiles of venetoclax in combination with decitabine and antifungals in plasma samples.

OUTLINE:

Participants receive decitabine intravenously (IV) over 1 hour on days 1-10 and may also receive decitabine on days 1-5 after achieving complete remission/complete remission with incomplete count recovery during consolidation/maintenance. Participants also receive venetoclax orally (PO) daily on days 1-28 of cycle 1 and on days 1-21 of subsequent cycles. Treatment repeat every 28 days for up to 24 cycles in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, participants are followed up every 3 to 6 months for up to 5 years.

Eligibility Criteria:
Inclusion Criteria:

* Patients with AML, BPDCN, biphenotypic or bilineage leukemia (including a myeloid component) or mixed phenotype acute leukemia (MPAL) who have failed prior therapy; patients with AML should have failed prior therapy or have relapsed after prior therapy; patients with isolated extramedullary AML are eligible
* Elderly (\> 60 year old) patients with newly diagnosed AML, BPDCN, or mixed phenotype acute leukemia (MPAL) not eligible for intensive chemotherapy
* Patients with newly diagnosed AML with poor risk complex karyotype and/or TP53 deletions/mutations equal or younger than 60 year old
* AML or BPDCN patients with prior history of MDS or CMML who received any therapy or no therapy for the MDS or CMML and progressed to AML, are eligible at the time of diagnosis of AML regardless of any prior therapy for MDS; the World Health Organization (WHO) classification will be used for AML
* Patients with high-risk MDS with bone marrow blasts between 10% and 20%, relapsed or refractory to prior hypomethylating agent (HMA) therapy, defined as prior receipt of 4 cycles of HMA therapy with failure to attain a response, or relapse after prior response to HMA therapy; patients with high risk chronic myelomonocytic leukemia (CMML) with bone marrow blasts \>= 10% regardless of prior therapy
* Eastern Cooperative Oncology Group (ECOG) performance status =\< 3
* White blood cell count =\< 10,000
* Adequate renal function including creatinine \< 2 unless related to the disease
* Adequate hepatic function including total bilirubin \< 2 x upper limit of normal (ULN) unless increase is due to Gilbert's disease or leukemic involvement
* Alanine aminotransferase (ALT) \< 3 x ULN unless considered due to leukemic involvement
* Provision of written informed consent
* Oral hydroxyurea and/or one dose of cytarabine (up to 2 g/m\^2) for patients with rapidly proliferative disease is allowed before the start of study therapy and while the patient is on active study treatment through cycle 1, as needed, for clinical benefit and after discussion with the principal investigator (PI); concurrent therapy for central nervous system (CNS) prophylaxis or continuation of therapy for controlled CNS disease is permitted
* Females must be surgically or biologically sterile or postmenopausal (amenorrheic for at least 12 months) or if of childbearing potential, must have a negative serum or urine pregnancy test within 72 hours before the start of the treatment
* Women of childbearing potential must agree to use an adequate method of contraception during the study and until 3 months after the last treatment; males must be surgically or biologically sterile or agree to use an adequate method of contraception during the study until 3 months after the last treatment

Exclusion Criteria:

* Patients having received any prior BCL2 inhibitor therapy
* Patients with t(15;17) karyotypic abnormality or acute promyelocytic leukemia (French-American-British \[FAB\] class M3-AML)
* Patients with symptomatic CNS leukemia or patients with poorly controlled CNS leukemia
* Active and uncontrolled comorbidities including active uncontrolled infection, uncontrolled hypertension despite adequate medical therapy, active and uncontrolled congestive heart failure New York Heart Association (NYHA) class III/IV, clinically significant and uncontrolled arrhythmia as judged by the treating physician
* Patients with known infection with human immunodeficiency virus (HIV) or active hepatitis B or C
* Any other medical, psychological, or social condition that may interfere with study participation or compliance, or compromise patient safety in the opinion of the investigator
* Pregnant or breastfeeding