Inclusion Criteria:  Patients 18-75 years of age with evidence of relapsed or refractory disease as defined by IMWG criteria and measurable disease as defined by any of the following:  Serum M-protein >= 0.5 g/dl (>= 10 g/l) Urine monoclonal protein >= 200 mg/24h Involved free light chain (FLC) level >= 10mg/dl (>= 100mg/l) and an abnormal serum free light chain ratio (< 0.26, or > 1.65)  Patients must have had at least 1 prior line of therapy including lenalidomide or cyclophosphamide, V or other PI, with or without ASCT  Patients with progressive disease (PD) as best response on V are excluded Patients with PD on D-based therapy may be eligible at the discretion of the treating physician  Refractory (progressed on or within 120 days of treatment) to their last treatment  Patients must be off last treatment for at least 2 weeks by the beginning of treatment on this protocol Hemoglobin >= 7g/dL Absolute neutrophil count (ANC) >= 1000/uL  Platelets >= 70,000/uL  If plasma cell percentage on bone marrow biopsy aspirate or core is > 30%, platelet requirement will be adjusted to 50,000/ul Total bilirubin < 1.5 mg/dL Aspartate aminotransferase (AST)/alanine aminotransferase (ALT)/alkaline phosphatase < 2 x the upper limit of normal (ULN) Serum creatinine < 2mg/dL or calculated creatinine clearance of >= 30ml/min using Modification of Diet in Renal Disease Study (MDRD) formula Left ventricular ejection fraction >= 30%; baseline echocardiogram (ECHO) is not required No uncontrolled arrhythmias No New York Heart Association class III-IV heart failure 12-lead electrocardiogram (ECG) with QT interval calculated by Fridericia Formula (QTcF) interval of =< 450 msec Patient must be able to swallow capsule or tablet Patients must provide informed consent Patients must have an Eastern Cooperative Oncology Group (ECOG) performance status (PS) score of < 2 Women of child bearing potential (WOCBP) must commit to either abstain continuously from heterosexual sexual intercourse or to use 2 methods of reliable birth control simultaneously. This includes one highly effective form of contraception (tubal ligation, intrauterine device [IUD], hormonal [birth control pills, injections, hormonal patches, vaginal rings or implants] or partner's vasectomy) and one additional effective contraceptive method (male latex or synthetic condom, diaphragm, or cervical cap). Contraception must begin 4 weeks prior to dosing and continue to 6 months after study treatment ending. Reliable contraception is indicated even where there has been a history of infertility, unless due to hysterectomy Investigators shall counsel WOCBP and male participants who are sexually active with WOCBP on the importance of pregnancy prevention and the implications of an unexpected pregnancy A negative pregnancy test will be required for all WOCBP within 24 hours before starting treatment drugs Breast feeding is not permitted  Male patients must agree to use an adequate method of contraception (latex or synthetic condom) for the duration of the study and up to 6 months after study treatment ending  Criteria also applies to azoospermic males Males should refrain from sperm donation during this time and continue for 6 months after study treatment ending  Exclusion Criteria:  Patients with active (untreated or relapsed) central nervous system (CNS) metastasis of myeloma  Patients with Waldenstrom macroglobulinemia, primary amyloid light-chain (AL) amyloidosis, primary plasma cell leukemia, or polyneuropathy, organomegaly, endocrinopathy, monoclonal gammopathy, and skin changes (POEMS) syndrome  Patients with secondary plasma cell leukemia are permitted Patients with peripheral neuropathy > National Cancer Institute (NCI) Common Terminology Criteria for Adverse Events (CTCAE) grade 2 Patients receiving concurrent corticosteroids at the time protocol therapy is initiated other than for physiologic maintenance treatment Concurrent use of complementary or alternative medicines that would confound the interpretation of toxicities and antitumor activity of the study drugs Patients with known allergies, hypersensitivity, or intolerance to panobinostat, daratumumab, or bortezomib  Unacceptable respiratory risk factors defined by any one of the following criteria:  Chronic obstructive pulmonary disease (COPD) with a forced expiratory volume in 1 second (FEV1) less than 50% of predicted normal Moderate or severe persistent asthma within the past 2 years, or currently has uncontrolled asthma of any classification  Unacceptable cardiac risk factors defined by any of the following criteria:  Patients with congenital long QT syndrome Any history of ventricular fibrillation or torsade de pointes Bradycardia defined as heart rate (HR) < 50 beats per minute (bpm) Left ventricular ejection fraction < 30% Patients who have received targeted or investigational agents within 2 weeks or within 5 half-lives of the agent and active metabolites (whichever is shorter) and who have not recovered from side effects of those therapies Patients who have undergone major surgery =< 2 weeks prior to starting study drug or who have not recovered from the side-effects of surgery Patients with known positivity for human immunodeficiency virus (HIV) or hepatitis C; baseline testing for HIV and hepatitis C is not required  Patients with active hepatitis B (defined as hepatitis B virus surface antigen positive [HBsAg+]); HBV screening is required prior to beginning therapy  Patients with prior hepatitis B vaccine are permitted (defined as hepatitis B virus surface antigen negative [HBsAg-], hepatitis B virus surface antibody positive [Anti-HBs+], hepatitis B virus core antibody negative (Anti-HBc-]) Non-active hepatitis B (HBsAg-, Anti-HBs+, hepatitis B virus core antibody positive [Anti-HBc+]) may only be enrolled following approval by the sponsor after consideration of risk of reactivation (additional screening and monitoring for hepatitis B and consultation with a liver disease specialist may be required)  Patients with a history of another primary malignancy that is currently clinically significant or currently requires active intervention, other than non-melanoma skin cancer and carcinoma in situ of the cervix, should not be enrolled  Patients are not considered to have a "currently active" malignancy if they have completed therapy for a prior malignancy, are disease free from a prior malignancy for >= 3 yrs, and are considered by their physician to be less than 30% risk of relapse Patients with a history of gastrointestinal surgery or other procedure that might, in the opinion of the investigator(s), interfere with the absorption or swallowing of the study drugs Patients with any significant history of non-compliance to medical regimens or unwilling or unable to comply with the instructions given to them by the study staff Any other medical condition, including mental illness or substance abuse, deemed by the investigator(s) to likely interfere with the patient's ability to sign informed consent, cooperate and participate in the study, or interfere with the interpretation of the results