Enzalutamide With Ribociclib in Treating Patients With Metastatic Castrate-Resistant, Chemotherapy Naive Prostate Cancer That Retains Retinoblastoma Expression

This phase Ib/II trial studies the safety, side effects, best dose, and effectiveness of ribociclib when given with enzalutamide in treating patients with castrate-resistant prostate cancer that has spread from the primary site (place where it started) to other places in the body (metastatic), is chemotherapy naive, and retains retinoblastoma expression. Testosterone can cause the growth of prostate cancer cells. Hormone therapy using enzalutamide may fight prostate cancer by blocking the use of testosterone by the tumor cells. Ribociclib may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Enzalutamide with ribociclib may be safe, tolerable and/or effective in treating metastatic, castrate-resistant, chemotherapy naive prostate cancer that retains retinoblastoma expression.

PRIMARY OBJECTIVES:

I. To determine the maximum tolerated dose of ribociclib in combination with 160 mg of enzalutamide. (Phase Ib) II. To determine efficacy with respect to the proportion of subjects that achieve a \>= 50% reduction in prostate-specific antigen (PSA) at 12 weeks. (Phase II)

SECONDARY OBJECTIVES:

I. PSA progression-free survival. II. Radiographic progression-free survival (rPFS). III. Safety. IV. Pharmacokinetics.

CORRELATIVE/EXPLORATORY/TERTIARY OBJECTIVES:

I. To evaluate the expression of retinoblastoma (RB) in circulating tumor cells (CTCs) and tumor tissue.

II. To evaluate other mechanisms of castrate resistance (such as androgen receptor \[AR\]-variant \[v\]7) in tumor tissue and CTCs.

III. To explore resistance mechanisms of cyclin dependent kinase (CDK)4/6 inhibitors in tumor samples in patients that progress on enzalutamide and ribociclib.

IV. Explore the use/correlation of circulating deoxyribonucleic acid (DNA)/exosomes in castrate-resistant prostate cancer (CRPC) patients treated with enzalutamide with and without ribociclib.

V. Androgen profiles and correlation to clinical outcomes. VI. Development of model explant systems to correlate with the clinical outcome.

OUTLINE: This is a phase I, dose-escalation study of ribociclib followed by a phase II study.

Patients receive enzalutamide orally (PO) once daily (QD) on days 1-28 and ribociclib PO QD on days 1-21 of each cycle. Cycles repeat every 28 days in the absence of disease progression or unacceptable toxicity. Patients also undergo blood sample collection, tumor biopsy, echocardiography (ECHO) or multigated acquisition scan (MUGA), bone scan, and computed tomography (CT) or magnetic resonance imaging (MRI) on study.

After completion of study treatment, patients are followed up every 3 months for 24 months.

Eligibility Criteria:
Inclusion Criteria:

* Willing and able to provide written informed consent and Health Insurance Portability and Accountability Act (HIPAA) authorization for the release of personal health information; NOTE: HIPAA authorization may be either included in the informed consent or obtained separately; consent and HIPPA authorization must be obtained prior to any screening procedures
* Histological or cytological proof of prostate cancer
* Documented progressive metastatic (m)CRPC based on at least one of the following criteria:

  * PSA progression defined as 25% increase over baseline value with an increase in the absolute value of at least 2 ng/mL that is confirmed by another PSA level with a minimum of a 1 week interval and a minimum PSA of 2 ng/mL
  * Soft-tissue progression defined as an increase \>= 20% in the sum of the longest diameter (LD) of all target lesions based on the smallest sum LD since treatment started or the appearance of one or more new lesions
  * Progression of bone disease (evaluable disease) or (new bone lesion\[s\]) by bone scan
* Have testosterone \< 50 ng/dL; patients must continue primary androgen deprivation with an luteinizing hormone-releasing hormone (LHRH) analogue (agonist or antagonist) if they have not undergone orchiectomy
* Eastern Cooperative Oncology Group (ECOG) performance status of 0-1
* Patients on long term (\> 6 months) anti-androgen therapy (e.g., flutamide, bicalutamide, nilutamide) will need to be off anti-androgen for 4 weeks (wash out period) and show evidence of disease progression off the anti-androgen; patients that have been on an anti-androgen 6 months or less will need to discontinue anti-androgen therapy prior to treatment start (no wash out period required)
* Absolute neutrophil count \>= 1.5 × 10\^9/L (obtained within 14 days prior to treatment start)
* Platelets (UNVPLT) \>= 100 x 10\^9/L (obtained within 14 days prior to treatment start)
* Hemoglobin (HGB) \>= 9 g/dl (obtained within 14 days prior to treatment start)
* Potassium (K) within normal limits for the institution or corrected to within normal limits with supplements before first dose of study medication (obtained within 14 days prior to treatment start)
* Total calcium (CA) (corrected for serum albumin) within normal limits for the institution or corrected to within normal limits with supplements before first dose of study medication (obtained within 14 days prior to treatment start)
* Magnesium within normal limits for the institution or corrected to within normal limits with supplements before first dose of study medication (obtained within 14 days prior to treatment start)
* Phosphorus within normal limits for the institution or corrected to within normal limits with supplements before first dose of study medication (obtained within 14 days prior to treatment start)
* International normalized ratio (INR) =\< 1.5 (obtained within 14 days prior to treatment start)
* Serum creatinine (CREAT) =\< 1.5 mg/dL or creatinine clearance \>= 50 mL/min (obtained within 14 days prior to treatment start)
* Alanine aminotransferase (ALT) and aspartate aminotransferase (AST) =\< 2.5 x upper limit of normal (ULN); if the patient has liver metastases, ALT and AST must still be =\< 2.5 x ULN; patients with liver metastases and AST/ALT above this limit will not be enrolled (obtained within 14 days prior to treatment start)
* Total serum bilirubin =\< 1.5 x ULN; or total bilirubin (TBILI) =\< 3.0 x ULN with direct bilirubin within normal range in patients with well documented Gilbert's syndrome (obtained within 14 days prior to treatment start)
* Men must agree to use adequate contraception prior to enrollment, for the duration of study participation and for at least 3 months thereafter
* Must be able to take oral medication without crushing, dissolving or chewing tablets

Exclusion Criteria:

* Prior exposure to abiraterone acetate or other specific cytochrome P450 (CYP)-17 inhibitors; abiraterone acetate given in the castration-sensitive setting is permissible if stopped at least 6 months prior to initial protocol treatment
* Prior exposure to enzalutamide, apalutamide, or other investigational AR directed therapy
* Prior chemotherapy for castration resistant disease; chemotherapy given in the castration-sensitive setting is permissible if stopped at least 4 weeks prior to treatment start
* Prior isotope therapy with strontium-89, samarium or radium-223 within 12 weeks of treatment start
* Administration of antifungal agents (itraconazole, fluconazole, etc) within 4 weeks of treatment start or unrecovered adverse events (AEs) due to agents administered more than 4 weeks of treatment start
* History of pituitary or adrenal dysfunction, active or symptomatic viral hepatitis or chronic liver disease
* Known symptomatic brain metastases
* Use of any prohibited concomitant medications: immunotherapy, 5 alpha reductase inhibitors, spironolactone, diethystilbestrol (DES), ketoconazole, newer medications targeting ARs; NOTE: the concurrent use of all other drugs, over-the-counter medications, or alternative therapies must be documented; the principal investigator should be alerted if the patient is taking any agent that interacts with CYP450 system
* Treatment-related toxicity from prior therapy \> grade 2
* Peripheral neuropathy \>= 2
* History of hypersensitivity to ribociclib or compounds of similar chemical or biologic composition to ribociclib including to peanut and soy or other drugs formulated with polysorbate 80; or enzalutamide
* Currently taking any herbal, alternative or food supplements (i.e., prostate cancer \[PC\]-Spes, saw palmetto, St John wort, etc.); all herbal, alternative and food supplements must be discontinued prior to treatment start; patients may continue on a daily multi-vitamin, calcium and vitamin D
* Planned surgery or radiation therapy during protocol treatment
* Hormonal-acting agents (including DES, aldosterone, and spironolactone but not including gonadotropin-releasing hormone \[GnRH\] agonists or antagonists) are forbidden during the trial and must be stopped prior to treatment start; no washout period will be required for any of these agents
* Initiation of bisphosphonate/denosumab therapy during protocol treatment; patients on stable doses of bisphosphonates or denosumab which have been started no less than 4 weeks prior to treatment start may continue on this medication; NOTE: initiation of bisphosphonate/denosumab therapy will be allowed for the treatment of osteoporosis or prevention of skeletal-related events (SRE) during protocol treatment
* Patient has a concurrent malignancy or malignancy within 3 years of treatment start, with the exception of adequately treated, basal or squamous cell carcinoma, non-melanomatous skin cancer or curatively resected cervical cancer
* Patient has impairment of gastrointestinal (GI) function or GI disease that may significantly alter the absorption of the study drugs (e.g., ulcerative diseases, uncontrolled nausea, vomiting, diarrhea, malabsorption syndrome, or small bowel resection)
* Patient has a known history of human immunodeficiency virus (HIV) infection (testing not mandatory)
* Patient has any other concurrent severe and/or uncontrolled medical condition that would, in the investigator's judgment, cause unacceptable safety risks, contraindicate patient participation in the clinical study or compromise compliance with the protocol (e.g., chronic pancreatitis, chronic active hepatitis, active untreated or uncontrolled fungal, bacterial or viral infections, etc.)
* Patient has clinically significant, uncontrolled heart disease and/or recent events including any of the following:

  * History of acute coronary syndromes (including myocardial infarction, unstable angina, coronary artery bypass grafting, coronary angioplasty, or stenting) or symptomatic pericarditis within 12 months prior to treatment start
  * History of documented congestive heart failure (New York Heart Association functional classification III-IV)
  * Documented cardiomyopathy
  * Patient has a left ventricular ejection fraction (LVEF) \< 50% as determined by multiple gated acquisition (MUGA) scan or echocardiogram (ECHO) at screening
  * History of any cardiac arrhythmias, eg., ventricular, supraventricular, nodal arrhythmias, or conduction abnormality within 12 months prior to treatment start
  * Family history of corrected QT interval (QTc) prolongation or of unexplainable sudden death at \< 50 years of age
  * On screening 12 lead electrocardiogram (ECG), any of the following cardiac parameters: bradycardia (heart rate \< 50 at rest), tachycardia (heart rate \> 90 at rest), PR interval \> 220 msec, QRS interval \> 109 msec, or Fridericia corrected QT (QTcF) \> 450 msec; congenital long QT syndrome or family history of long QT syndrome
  * Systolic blood pressure (SBP) \> 160 mmHg or \< 90 mmHg
  * Bradycardia (heart rate \< 50 at rest), by ECG or pulse, at screening
* On screening, inability to determine the QTcF interval on the ECG (i.e.: unreadable or not interpretable) or QTcF \> 450 msec (using Fridericia's correction); all as determined by screening ECG (mean of triplicate ECGs)
* Patient is currently receiving any of the following medications and cannot be discontinued 7 days prior to treatment start:

  * Known strong inducers or inhibitors of cytochrome P450, family 3, subfamily A, polypeptide 4 (CYP3A4)/cytochrome P450, family 3, subfamily A, polypeptide 4 (5), including grapefruit, grapefruit hybrids, pummelos, star-fruit, and Seville oranges
  * That have a narrow therapeutic window and are predominantly metabolized through CYP3A4/5
  * That have a known risk to prolong the QT interval or induce Torsades de Pointes
  * Herbal preparations/medications, dietary supplements
* Patient is currently receiving or has received systemic corticosteroids within \< 2 weeks prior to treatment start, or who have not fully recovered from side effects of such treatment

  \* The following uses of corticosteroids are permitted: a short duration (\< 5 days) of systemic corticosteroids; any duration of topical applications (e.g. for rash), inhaled sprays (e.g., for obstructive airways diseases), eye drops or local injections (e.g., intra-articular)
* Patient is currently receiving warfarin or other coumarin-derived anticoagulant for treatment, prophylaxis or otherwise; therapy with heparin, low molecular weight heparin (LMWH) or fondaparinux is allowed
* PatientParticipation in other studies involving investigational drug(s) within 30 days prior to randomization or within 5 half-lives of the investigational product (whichever is longer) or participation in any other type of medical research judged not to be scientifically or medically compatible with this study. If the patient is enrolled or planned to be enrolled in another study that does not involve an investigational drug, the agreement of the Novartis study medical lead is required to establish eligibility
* Patient who has received radiotherapy =\< 4 weeks or limited field radiation for palliation =\< 2 weeks prior to treatment start, and who has not recovered to grade 1 or better from related side effects of such therapy (exceptions include alopecia) and/or in whom \>= 30% of the bone marrow was irradiated
* Patients with central nervous system (CNS) involvement unless they meet ALL of the following criteria:

  * At least 4 weeks from prior therapy completion (including radiation and/or surgery) to treatment start
  * Clinically stable CNS tumor at the time of screening and not receiving steroids and/or enzyme-inducing anti-epileptic medications for brain metastases
* Patient has had major surgery within 14 days prior to treatment start or has not recovered from major side effects (tumor biopsy is not considered as major surgery)
* Patient has not recovered from all toxicities related to prior anticancer therapies to National Cancer Institute (NCI)-Common Terminology Criteria for Adverse Events (CTCAE) version 4.03 grade \< 1 (exception to this criterion: patients with grade 1 taxane-induced neuropathy, any grade of alopecia, amenorrhea or other toxicities not considered a safety risk for the patient as per investigator's discretion, are allowed to enter the study)
* Patient with a Child-Pugh score B or C
* Patient has a history of non-compliance to medical regimen or inability to grant consent
* Sexually active males unless they use a condom during intercourse while taking the drug and for 30 days after stopping treatment and should not father a child in this period; a condom is required to be used by vasectomized men in order to prevent delivery of the drug via seminal fluid