Inclusion Criteria:  Patient or legally acceptable representative must be able to understand and willing to sign an IRB approved written informed consent document. Patients must be 18 years or older Patients must have histologically confirmed metastatic pancreatic adenocarcinoma. Patients must have radiographically measurable disease defined as non-radiated lesions that can be accurately measured in at least one dimension (longest diameter to be recorded) as greater than or equal to 10mm with calipers by radiographic exam.Further details listed in section 5.5. Patients must have received 5FU or Gemcitabine based first line treatment (in combination with platinum, topoisomerase I inhibitor, or nab-paclitaxol) for metastatic disease with evidence of treatment failure or intolerance. If a subject received therapy in the adjuvant/neoadjuvant setting: Tumor recurrence must have occurred no sooner than 6 months after completion of the last dose of therapy. Radiation therapy for the locally advanced disease is allowed The patient must have a life expectancy of minimum 3 months.  Patient must have normal bone marrow and organ function as defined below:  Absolute neutrophil count >1,500/mcl Platelets >100,000/mcl Hemoglobin >9.0 g/dL Creatinine should be below the upper limit of normal OR Creatinine clearance >60 mL/min/1.73 m2 for patients with creatinine levels above institutional normal Patient not on anticoagulation must have international Normalized Ratio (INR) and activated partial thromboplastin time (PTT) <1.5 upper limit of normal. Patient will have an ECOG performance status of 0, 1, or 2 Patient must consent for baseline and on treatment biopsies, if prior baseline biopsies have been performed, and stored within the University of Rochester Cancer Library, then baseline biopsies are not necessary. Baseline biopsy must be of a lesion post-relapse. Patients with biliary stents for obstruction may be included in the study Female subjects of childbearing potential must demonstrate a negative urine or serum pregnancy test. Women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control, abstinence) prior to study entry and for the duration of the study treatment. Should a woman become pregnant or suspect she is pregnant while participating in this study, she must inform her treating physician immediately. Male subjects with a female partner of child-bearing potential must agree to use 2 adequate methods of contraception (barrier + hormonal for example).  Exclusion Criteria:  Patient with any histologic variant of PDAC such as adenosquamous, acinar cell, duodenal adenocarcinoma, or ampullary adenocarcinoma Patient has a history of other malignancy ≤ 3 years previous with the exception of basal cell or squamous cell carcinoma of the skin which were treated with local resection only or carcinoma in situ of the cervix. Patient who is receiving or has received an investigational immunotherapy, investigational combinations of cytotoxic chemotherapy such as liposomal irinotecan are permitted Patient has a history of allergic reactions attributed to compounds of similar chemical or biologic composition to Pepinemab (VX15/2503) or Avelumab Patient with uncontrolled intercurrent illness including, but not limited to, ongoing or active infection symptomatic congestive heart failure, unstable angina pectoris, cardiac arrhythmia, any clinically active malabsorption syndrome, inflammatory bowel disease, any condition that increases the risk of severe irinotecan gastrointestinal toxicity, or psychiatric illness/social situations that would limit compliance with study requirements Has an active autoimmune disease, or a documented history of autoimmune disease or a syndrome that requires steroids or immunosuppressive agents. Subjects with vitiligo or resolved childhood asthma/atopy would be exceptions to this rule. Has had an allogenic tissue/solid organ transplant. Has interstitial lung disease or a history of pneumonitis that required oral or intravenous glucocorticoids to assist with management. Has received or will receive a live vaccine within 30 days prior to the first administration of study medication. Flu vaccines that do not contain live virus are permitted. Has known active Hepatitis B or C. Patient who is pregnant and/or breastfeeding. Persisting toxicity related to prior anti-cancer therapy Major surgery or use of investigational drug within the 28 days before the start of treatment Current alcohol or drug abuse Prior organ or bone marrow transplantation Inability to comply with visit schedule or other protocol requirements including mandatory study biopsies