A method for performing an AllReduce processing task by a plurality of processing units, wherein the processing task is divided into a plurality of portions, each portion being processed by a processing unit of the plurality of processing units, each portion comprising a group of operations that are to be performed at a processing unit of the plurality of processing units, the group of operations comprising operations of a first type and operations of a second type, the method comprising: building, at each processing unit, a first queue for performing the operations of the first type and a second queue for performing the operations of the second type; obtaining a dependency relationship between a group of operations that are to be performed at a processing unit of the plurality of processing units and a group of operations that are to be performed at other processing units among the plurality of processing units; and performing operations in the first queue and operations in the second queue respectively at a processing unit based on the dependency relationship; loading, to a memory of the processing unit, a first code for performing a first group of operations of the first type and a second code for performing a second group of operations of the second type; wherein performing operations in the first queue and operations in the second queue respectively at the processing unit based on the dependency relationship comprises: performing the operations in the first queue based on first code; and performing the operations in the second queue based on second code; wherein the operations in the first queue and the operations in the second queue are performed in parallel. wherein the processing task is divided into a plurality of portions, each portion being processed by a processing unit of the plurality of processing units, each portion comprising a group of operations that are to be performed at a processing unit of the plurality of processing units, the group of operations comprising operations of a first type and operations of a second type, the method comprising: building, at each processing unit, a first queue for performing the operations of the first type and a second queue for performing the operations of the second type; obtaining a dependency relationship between a group of operations that are to be performed at a processing unit of the plurality of processing units and a group of operations that are to be performed at other processing units among the plurality of processing units; and performing operations in the first queue and operations in the second queue respectively at a processing unit based on the dependency relationship; loading, to a memory of the processing unit, a first code for performing a first group of operations of the first type and a second code for performing a second group of operations of the second type; wherein performing operations in the first queue and operations in the second queue respectively at the processing unit based on the dependency relationship comprises: performing the operations in the first queue based on first code; and performing the operations in the second queue based on second code; wherein the operations in the first queue and the operations in the second queue are performed in parallel. building, at each processing unit, a first queue for performing the operations of the first type and a second queue for performing the operations of the second type; obtaining a dependency relationship between a group of operations that are to be performed at a processing unit of the plurality of processing units and a group of operations that are to be performed at other processing units among the plurality of processing units; and performing operations in the first queue and operations in the second queue respectively at a processing unit based on the dependency relationship; loading, to a memory of the processing unit, a first code for performing a first group of operations of the first type and a second code for performing a second group of operations of the second type; wherein performing operations in the first queue and operations in the second queue respectively at the processing unit based on the dependency relationship comprises: performing the operations in the first queue based on first code; and performing the operations in the second queue based on second code; wherein the operations in the first queue and the operations in the second queue are performed in parallel. performing the operations in the first queue based on first code; and performing the operations in the second queue based on second code; wherein the operations in the first queue and the operations in the second queue are performed in parallel.