An analog neural memory system (3400, 3500, 3700) comprising: a first vector-by-matrix multiplication array (1300, 3401, 3501, 3701) comprising an array of non-volatile memory cells organized into rows and columns, wherein each memory cell (210, 610, 710) comprises a bit line terminal, a source line terminal, and a word line terminal; a second vector-by-matrix multiplication array (1300, 3402, 3502, 3702) comprising an array of non-volatile memory cells organized into rows and columns, wherein each memory cell (210, 610, 710) comprises a bit line terminal, a source line terminal, and a word line terminal; a first plurality of bit lines (BLO, BL1, BL2, BL3, 3421, 3521, 3721) coupled to the bit line terminals of columns of memory cells in the first vector-by-matrix multiplication array; a second plurality of bit lines (BLO, BL1, BL2, BL3, 3422, 3522, 3722) coupled to the bit line terminals of columns of memory cells in the second vector-by-matrix multiplication array; a first low voltage row decoder (3405, 3505, 3705) coupled to the word line terminals of rows of memory cells in the first vector-by-matrix multiplication array; a second low voltage row decoder (3406, 3506, 3706) coupled to the word line terminals of rows of memory cells in the second vector-by-matrix multiplication array; a first plurality of word lines (WLA0, WLB0, WLA1, WLB2, WLA2, WLB2, WLA3, WLB3, 3411, 3511, 3711) coupled to the word line terminals of rows of memory cells in the first vector-by-matrix multiplication array by the first low voltage row decoder; a second plurality of word lines (WLA0, WLB0, WLA1, WLB2, WLA2, WLB2, WLA3, WLB3, 3412, 3512, 3712) coupled to the word line terminals of rows of memory cells in the second vector-by-matrix multiplication array by the second low voltage row decoder; and a plurality of source lines (SLO, SL1, 3434, 3533, 3733, 5133), wherein each of the plurality of source lines is coupled to the source line terminals of rows of memory cells in the first vector-by-matrix multiplication array and to the source line terminals of rows of memory cells in the second vector-by-matrix multiplication array; wherein the first plurality of word lines and the second plurality of word lines are parallel to the first plurality of bit lines and the second plurality of bit lines and perpendicular to the plurality of source lines, and wherein during a neuron read operation of the first vector-by-matrix multiplication array or the second vector-by-matrix multiplication array an output is provided on the plurality of source lines that equals a sum of products generated for each cell in the respective first vector-by-matrix multiplication array or second vector-by-matrix multiplication array, wherein each product is equal to an input current received on a word line terminal of a cell multiplied by a weight stored in the cell. a first vector-by-matrix multiplication array (1300, 3401, 3501, 3701) comprising an array of non-volatile memory cells organized into rows and columns, wherein each memory cell (210, 610, 710) comprises a bit line terminal, a source line terminal, and a word line terminal; a second vector-by-matrix multiplication array (1300, 3402, 3502, 3702) comprising an array of non-volatile memory cells organized into rows and columns, wherein each memory cell (210, 610, 710) comprises a bit line terminal, a source line terminal, and a word line terminal; a first plurality of bit lines (BLO, BL1, BL2, BL3, 3421, 3521, 3721) coupled to the bit line terminals of columns of memory cells in the first vector-by-matrix multiplication array; a second plurality of bit lines (BLO, BL1, BL2, BL3, 3422, 3522, 3722) coupled to the bit line terminals of columns of memory cells in the second vector-by-matrix multiplication array; a first low voltage row decoder (3405, 3505, 3705) coupled to the word line terminals of rows of memory cells in the first vector-by-matrix multiplication array; a second low voltage row decoder (3406, 3506, 3706) coupled to the word line terminals of rows of memory cells in the second vector-by-matrix multiplication array; a first plurality of word lines (WLA0, WLB0, WLA1, WLB2, WLA2, WLB2, WLA3, WLB3, 3411, 3511, 3711) coupled to the word line terminals of rows of memory cells in the first vector-by-matrix multiplication array by the first low voltage row decoder; a second plurality of word lines (WLA0, WLB0, WLA1, WLB2, WLA2, WLB2, WLA3, WLB3, 3412, 3512, 3712) coupled to the word line terminals of rows of memory cells in the second vector-by-matrix multiplication array by the second low voltage row decoder; and a plurality of source lines (SLO, SL1, 3434, 3533, 3733, 5133), wherein each of the plurality of source lines is coupled to the source line terminals of rows of memory cells in the first vector-by-matrix multiplication array and to the source line terminals of rows of memory cells in the second vector-by-matrix multiplication array; wherein the first plurality of word lines and the second plurality of word lines are parallel to the first plurality of bit lines and the second plurality of bit lines and perpendicular to the plurality of source lines, and wherein during a neuron read operation of the first vector-by-matrix multiplication array or the second vector-by-matrix multiplication array an output is provided on the plurality of source lines that equals a sum of products generated for each cell in the respective first vector-by-matrix multiplication array or second vector-by-matrix multiplication array, wherein each product is equal to an input current received on a word line terminal of a cell multiplied by a weight stored in the cell.