Alisertib in Treating Patients With Myelofibrosis or Relapsed or Refractory Acute Megakaryoblastic Leukemia

The purpose of this study is to evaluate the safety of alisertib and its effect, bad and/or good, on acute megakaryoblastic leukemia (AMKL) or myelofibrosis (MF). The study drug, alisertib, is an investigational drug. An investigational drug is one that has not been approved by the U.S. Food and Drug Administration (FDA). Alisertib has shown evidence in the lab that it may have an effect on a type of cell that produces platelets. This cell is called a megakaryocyte and it is known to be defective (doesn't work well) in both AMKL and MF.

PRIMARY OBJECTIVES:

I. Determine the safety profile of alisertib in patients with acute megakaryoblastic leukemia (AMKL) and in patients with myelofibrosis (MF).

SECONDARY OBJECTIVES:

I. Determine preliminary efficacy of alisertib in both populations.

TERCIARY OBJECTIVES:

I. Describe pharmacodynamics (PD) effects of alisertib in peripheral blood and/or bone marrow samples.

II. Evaluate the relationship between biomarker expression levels and response to alisertib.

III. Evaluate reduction in splenomegaly by palpation (MF arm only). IV. Evaluate improvement in MF symptoms (MF arm only), as assessed by the Myeloproliferative Neoplasm Symptom Assessment form (MPN-SAF).

V. Assess change in bone marrow fibrosis in patients in the MF arm.

OUTLINE:

Patients receive alisertib orally (PO) twice daily (BID) on days 1-7. Courses repeat every 21 days in the absence of disease progression or unacceptable toxicity.

After completion of study treatment, patients are followed up at approximately 30 days and 6 months.

Eligibility Criteria:
Inclusion Criteria:

* AMKL PATIENTS: Patients must have a confirmed diagnosis of relapsed/refractory acute megakaryoblastic leukemia (AMKL), as defined by World Health Organization (WHO) criteria
* AMKL PATIENTS: Patients must have an Eastern Cooperative Oncology Group (ECOG) status 0-2
* AMKL PATIENTS: Total bilirubin =\< 1.5 x upper limit of normal (ULN)
* AMKL PATIENTS: Aspartate aminotransferase (AST)/alanine aminotransferase (ALT) =\< 2.5 x ULN
* AMKL PATIENTS: Creatinine \< 1.5 x ULN or calculated creatinine clearance \> 30 ml/min
* AMKL PATIENTS: Prothrombin time (PT) and partial thromboplastin time (PTT) =\< 1.5 x ULN
* AMKL PATIENTS: Absolute neutrophil count (ANC) \>= 1500/mm\^3
* AMKL PATIENTS: Platelets \>= 100,000/mm\^3
* AMKL PATIENTS: Hemoglobin \> 9 g/dL
* AMKL PATIENTS: Patients must have estimated life expectancy of 6 months or greater
* AMKL PATIENTS: Female patients of child-bearing potential (FOCBP) must have a negative serum beta-human chorionic gonadotropin (HCG) pregnancy test within 7 days prior to registration; NOTE: a FOCBP is any woman (regardless of sexual orientation, having undergone a tubal ligation, or remaining celibate by choice) who meets the following criteria:

  * Has not undergone a hysterectomy or bilateral oophorectomy
  * Has had menses at any time in the preceding 12 consecutive months (and therefore has not been naturally postmenopausal for \> 12 months)
* AMKL PATIENTS: Female patients must meet at least one of the following conditions:

  * Must be post-menopausal for at least 1 year prior to registration (not of childbearing potential)
  * Must be surgically sterilized
  * Willing to use an acceptable method of birth control (i.e. hormonal contraceptive, intra-uterine device, diaphragm with spermicide, condom with spermicide, or abstinence) for the duration of the study
* AMKL PATIENTS: Male patients, even if surgically sterilized (status post-vasectomy) agrees to use an acceptable method for contraception during the entire study treatment period through 4 months after the last dose of alisertib
* AMKL PATIENTS: Patients must be able to understand and willing to sign a written informed consent
* MF PATIENTS: Patients must have a confirmed diagnosis of myelofibrosis (MF), as defined by WHO criteria
* MF PATIENTS: Patients must be intermediate I risk or beyond and meet the following:

  * In need of treatment
  * Intolerant or refractory to ruxolitinib (or other investigational Janus kinase \[JAK\]-inhibitors) OR unlikely to benefit from ruxolitinib
  * Ineligible for stem cell transplantation
* MF PATIENTS: Patients must have an ECOG status 0-2
* MF PATIENTS: Direct bilirubin \< 1.5 x ULN
* MF PATIENTS: ALT/AST =\< 2.5 x ULN
* MF PATIENTS: Creatinine \< 1.5 x ULN or calculated creatinine clearance \> 30 ml/min
* MF PATIENTS: PT and PTT =\< 1.5 x ULN
* MF PATIENTS: ANC \>= 1500/mm\^3
* MF PATIENTS: Platelets \>= 100,000/mm\^3
* MF PATIENTS: Patients must have estimated life expectancy of 6 months or greater
* MF PATIENTS: Female patients of child-bearing potential (FOCBP) must have a negative serum beta-HCG pregnancy test within 7 days prior to registration
* MF PATIENTS: Female patients must meet at least one of the following conditions:

  * Must be post-menopausal for at least 1 year prior to registration (not of childbearing potential)
  * Must be surgically sterilized
  * Willing to use an acceptable method of birth control (i.e. hormonal contraceptive, intra-uterine device, diaphragm with spermicide, condom with spermicide, or abstinence) for the duration of the study
* MF PATIENTS: Male patients, even if surgically sterilized (i.e. status post-vasectomy) agrees to use an acceptable method for contraception during the entire study treatment period through 4 months after the last dose of alisertib
* MF PATIENTS: Patients must be able to understand and willing to sign a written informed consent

Exclusion Criteria:

* Patients who have received treatment with clinically significant enzyme inducers (such as enzyme inducing antiepileptic drugs phenytoin, carbamazepine, or phenobarbital, or rifampin, rifabutin, rafapentine, or St. John's wort) within 14 days prior to registration are not eligible
* Patients who have received any investigational products, antineoplastic therapies, or radiotherapy within 14 days prior to registration are not eligible; NOTE: patients actively receiving hydroxyurea are eligible and may continue to receive hydroxyurea through cycle 1 of protocol treatment
* Patients who have received prior administration of an Aurora A kinase targeted agent (including alisertib) are not eligible
* Patients who have received corticosteroids within 7 days prior registration are not eligible, UNLESS the patient has been taking a continuous dose of no more than 15 mg/day of prednisone for at least 1 month prior; NOTE: low dose steroid use for control of nausea and vomiting will be allowed; topical steroid use and inhaled steroids are also permitted
* Patients who are candidates (eligible and willing) for standard and/or potentially curative treatments are not eligible
* Patients who have had received radiation therapy to more than 25% of the bone marrow are not eligible (whole pelvic radiation is considered to be over 25%)
* Patients who have had major surgery within one month (28 days) prior to registration are not eligible
* Patients who have had prior allogenic bone marrow or organ transplantation are not eligible
* Patients who have had grade 2 or higher diarrhea, despite optimal antidiarrheal supportive care, within 7 days prior to registration are not eligible
* Patients who have had grade 2 or higher peripheral neuropathy within 14 days prior to registration are not eligible
* Patients who have had a myocardial infarction within 6 months (24 weeks) prior to registration are not eligible
* Patients who have class III or IV heart failure (as defined by the New York Heart Association), uncontrolled angina, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia or active conduction system abnormalities are not eligible
* Patients who have known gastrointestinal (GI) disease or GI procedures which could interfere with the oral absorption or tolerance of alisertib are not eligible; examples include (but are not limited to) partial gastrectomy, history of small intestine surgery, and celiac disease
* Patients who have a known history of uncontrolled sleep apnea syndrome and other conditions that could result in excessive daytime sleepiness (such as severe chronic obstructive pulmonary disease or requirement for supplemental oxygen) are not eligible
* Patients who have a requirement for constant administration of proton pump inhibitor, histamine-2 (H2) antagonist, or pancreatic enzymes are not eligible; intermittent usage of antacids or H2 antagonists are allowed
* Patients who have a systemic infection requiring intravenous (IV) antibiotic therapy within 14 days prior to registration (or other severe infection) are not eligible
* Patients who are known human immunodeficiency virus (HIV) positive are not eligible
* Patients who are known hepatitis B surface antigen positive are not eligible
* Patients who have known or suspected active hepatitis C infections are not eligible; NOTE: patients who are hepatitis C surface antigen positive are eligible
* Female patients who are pregnant or breast feeding are not eligible
* Patients who have a severe acute or chronic medical or psychiatric condition, including uncontrolled diabetes, malabsorption, resection of the pancreas or upper small bowel, requirement for pancreatic enzymes, any condition that would modify small bowel absorption of oral medications, or laboratory abnormality that may increase the risk associated with study participation or investigational product administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for enrollment in this study are not eligible
* Patients who have symptomatic central nervous system (CNS) involvement are not eligible
* Patients who have been diagnosed or treated for another malignancy within 3 years prior to registration are not eligible aside from these exceptions: completely resected basal cell carcinoma or squamous cell carcinoma of the skin, an in situ malignancy, or low-risk prostate cancer after curative therapy
* Patients who are unable to swallow oral medication or are unwilling to comply with the administration requirements are not eligible
* Patients who require administration of myeloid growth factors or platelet transfusions within 14 days prior to registration are not eligible