An apparatus for processing sparse matrices, comprising: a plurality of graphic processing units each comprising a sparse matrix processing mechanism including: a scheduler being included in a processing element of the graphic processing unit, wherein the processing element further comprises: a logic configured to read operands included in instructions related to multiplying the sparse matrices; a multiplication unit configured to multiply two or more operands; and a write result logic configured to write a result of the multiplication of the two or more operands at the multiplication unit; wherein the scheduler is configured to retrieve stored operand values upon the operands being read at the logic, wherein the scheduler is further configured to identify operands having a zero value in the sparse matrices being processed and prevent scheduling of the operands having the zero value at the multiplication unit and schedule non-zero operands at the multiplication unit, wherein the scheduler is further configured to write a zero value to the write result logic for operands having a zero value, characterized in that the sparse matrix processing mechanism further comprises: a sparse pattern tracker configured to determine sparsity patterns of the sparse matrices; and a compression logic configured to compress the sparse matrices, wherein the compressed sparse matrices are dynamically generated from the sparse matrices, based on sparse indices, wherein the sparse indices are defined as percent of non-zero entries of the sparse matrices according to the determined sparsity patterns; a sparse compressed buffer to store the compressed sparse matrices, wherein the compression logic is further configured to decompress the compressed sparse matrices into the sparse matrices prior to processing by the one or more processing units. a plurality of graphic processing units each comprising a sparse matrix processing mechanism including: a scheduler being included in a processing element of the graphic processing unit, wherein the processing element further comprises: a logic configured to read operands included in instructions related to multiplying the sparse matrices; a multiplication unit configured to multiply two or more operands; and a write result logic configured to write a result of the multiplication of the two or more operands at the multiplication unit; wherein the scheduler is configured to retrieve stored operand values upon the operands being read at the logic, wherein the scheduler is further configured to identify operands having a zero value in the sparse matrices being processed and prevent scheduling of the operands having the zero value at the multiplication unit and schedule non-zero operands at the multiplication unit, wherein the scheduler is further configured to write a zero value to the write result logic for operands having a zero value, a scheduler being included in a processing element of the graphic processing unit, wherein the processing element further comprises: a logic configured to read operands included in instructions related to multiplying the sparse matrices; a multiplication unit configured to multiply two or more operands; and a write result logic configured to write a result of the multiplication of the two or more operands at the multiplication unit; a logic configured to read operands included in instructions related to multiplying the sparse matrices; a multiplication unit configured to multiply two or more operands; and a write result logic configured to write a result of the multiplication of the two or more operands at the multiplication unit; wherein the scheduler is configured to retrieve stored operand values upon the operands being read at the logic, wherein the scheduler is further configured to identify operands having a zero value in the sparse matrices being processed and prevent scheduling of the operands having the zero value at the multiplication unit and schedule non-zero operands at the multiplication unit, wherein the scheduler is further configured to write a zero value to the write result logic for operands having a zero value, characterized in that the sparse matrix processing mechanism further comprises: a sparse pattern tracker configured to determine sparsity patterns of the sparse matrices; and a compression logic configured to compress the sparse matrices, wherein the compressed sparse matrices are dynamically generated from the sparse matrices, based on sparse indices, wherein the sparse indices are defined as percent of non-zero entries of the sparse matrices according to the determined sparsity patterns; a sparse compressed buffer to store the compressed sparse matrices, wherein the compression logic is further configured to decompress the compressed sparse matrices into the sparse matrices prior to processing by the one or more processing units. a sparse pattern tracker configured to determine sparsity patterns of the sparse matrices; and a compression logic configured to compress the sparse matrices, wherein the compressed sparse matrices are dynamically generated from the sparse matrices, based on sparse indices, wherein the sparse indices are defined as percent of non-zero entries of the sparse matrices according to the determined sparsity patterns; a sparse compressed buffer to store the compressed sparse matrices, wherein the compression logic is further configured to decompress the compressed sparse matrices into the sparse matrices prior to processing by the one or more processing units.