Inclusion Criteria:  All subjects must have the ability to understand and the willingness to sign a written informed consent Age 18 and older Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 (corresponds to Karnofsky Performance Status [KPS] of >= 70) Life expectancy greater than 16 weeks Relapsed/refractory disease; patients must have failed >= 2 prior systemic therapies and have no treatment options known to provide clinical benefit  Biopsy confirmed relapsed or refractory B-cell lymphoma of the following subtypes; patients with a partial response to a previous treatment are allowed.  Follicular grade 1 or 2, or 3 marginal zone or small lymphocytic lymphoma Diffuse large B-cell lymphoma, non-germinal center B-cell like (GCB) type determined by immunohistochemistry using Han's algorithm Mantle cell lymphoma Patients must have at least two separate tumor sites, one of which is at least 2 cm in diameter and peripherally accessible and amenable for intratumoral injection of CAS3/SS3 with stabilization by palpation, and the other is at least 1.5 cm in diameter Tumor specimens must be available for immunological studies either from a previous biopsy or a new biopsy obtained before the initiation of the study. Paraffin-embedded specimens are acceptable  Required wash out periods for prior therapy:  Topical therapy: 2 weeks Chemotherapy: 4 weeks Radiotherapy: 4 weeks Other investigational therapy: 4 weeks Monoclonal antibody or immunotherapy: 4 weeks Targeted therapies: 4 weeks Allogeneic transplant: 6 months  Absolute neutrophil count (ANC) >= 1,000/mm^3  NOTE: Growth factor is not permitted within 14 days of ANC assessment unless cytopenia is secondary to disease involvement  Platelets >= 75,000/mm^3  NOTE: Platelet transfusions are not permitted within 14 days of platelet assessment unless cytopenia is secondary to disease involvement Hemoglobin >= 8 g/dl Total serum bilirubin (mg/dL): =< 1.5 x upper normal limit (ULN) Aspartate aminotransferase (AST) < 3 x ULN without liver metastasis and < 5 x ULN with liver metastasis Alanine aminotransferase (ALT) < 3 x ULN without liver metastasis and < 5 x ULN with liver metastasis Adequate renal function as determined by serum creatinine =< 2.0 mg/dL or creatinine clearance of >= 50 mL/min per the Cockcroft-Gault formula International normalized ratio (INR) OR prothrombin (PT) =< 1.5 x ULN Activated partial thromboplastin time (aPTT) =< 1.5 x ULN Female of childbearing potential: serum pregnancy test Cardiac function 12-lead electrocardiogram (ECG) to confirm the absence of clinically significant arrhythmias Left ventricular ejection fraction (LVEF) >= 45% Oxygen saturation on room air of >= 92% The effects of CAS3/SS3 on the developing fetus are unknown. For this reason, women of child-bearing potential and men must agree to use adequate contraception (hormonal or barrier method of birth control or abstinence) prior to study entry and for six months following duration of study participation. Should a woman become pregnant or suspect that she is pregnant while participating on the trial, she should inform her treating physician immediately  Exclusion Criteria:  Patients may not be receiving any other investigational agents, or concurrent biological, chemotherapy, or radiation therapy Current use of anticoagulant therapy (aspirin [ASA] =< 325 mg per day allowed) or history of significant bleeding diathesis Treatment with corticosteroids or other systemic immunosuppressive medication (e.g., methotrexate, rapamycin) within 28 days of study treatment. Note: patients with adrenal insufficiency may take up to 7.5 mg of prednisone or equivalent daily. Topical and inhaled corticosteroids in standard doses are allowed Patients with rapid progression of disease requiring immediate anti lymphoma therapy Patients should not have any uncontrolled illness including ongoing or active infection Pregnant or lactating women are excluded from this study because CAS3/SS3is an agent with the potential for teratogenic or abortifacient effects. Because there is an unknown but potential risk for adverse events in nursing infants secondary to treatment of the mother with CAS3/SS3, breastfeeding should be discontinued if the mother is treated with CAS3/SS3 Prior malignancy (active within 5 years of screening) except basal cell or completely excised non-invasive squamous cell carcinoma of the skin, or in situ squamous cell carcinoma of the cervix Pre-existing autoimmune or antibody mediated disease including: systemic lupus erythematosus, rheumatoid arthritis, multiple sclerosis, Sjogren's syndrome, autoimmune thrombocytopenia, Addison's disease, but excluding the presence of autoantibodies without clinical autoimmune disease History of inflammatory bowel disease (e.g., Crohn's disease or ulcerative colitis), celiac disease, or other chronic gastrointestinal conditions associated with diarrhea, or current acute colitis of any origin History of diverticulitis (even a single episode) or evidence of diverticulitis at baseline, including evidence limited to computed tomography (CT) scan only. Note: diverticulosis is not an exclusion criterion per se Graft versus host disease Severe psoriasis Active thyroiditis History of uveitis Known history of human immunodeficiency virus (HIV) with detectable viral load or patients with acquired immuno-deficiency syndrome (AIDS) are excluded Known active or chronic viral hepatitis B or C infection Patients with active infection or with a fever > 38.5 degrees Celsius (C) within three days prior to the first scheduled treatment Active central nervous system (CNS) metastases History of allergic reactions attributed to compounds of similar composition to agatolimod sodium (PF-3512676) or tremelimumab History of allergic reactions attributed to compounds of similar chemical or biologic composition to CAS3/SS3 Significant cardiovascular disease (i.e. New York Heart Association [NYHA] >= class 3 congestive heart failure; myocardial infarction within the past 6 months; unstable angina; coronary angioplasty within the past 6 months; uncontrolled atrial or ventricular cardiac arrhythmias) Any other condition that would, in the Investigator's judgment, contraindicate the patient's participation in the clinical study due to safety concerns with clinical study procedures Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics)