A banknote processing device that sorts and stacks plural banknotes, the banknote processing device comprising: a take-in unit that is provided at an upper position of the banknote processing device and takes in the plural banknotes placed on the take-in unit one by one; a recognition unit that recognizes the banknote taken in by the take-in unit; a first upper stacking unit that is provided at an upper position of the banknote processing device; a second upper stacking unit that is provided adjacent to the first upper stacking unit; first to fourth lower stacking units (106a-106d) that are arranged in parallel at lower positions of the banknote processing device and stack banknotes of at least one specific denomination, and each of the first to fourth lower stacking units (106a-106d) includes an opening that opens on a front side of the banknote processing device; a transport unit that transports the banknote taken in by the take-in unit to one of the first and second upper stacking units (102a, 102b) and the first to fourth lower stacking units (106a-106d); and a controller that controls the transport unit to transport the banknote to the first upper stacking unit or the second upper stacking unit when a recognition result obtained by the recognition unit regarding the banknote is a rejected banknote, and to transport the banknote to one of the first to fourth lower stacking units (106a-106d) when the recognition result obtained by the recognition unit regarding the banknote is not a rejected banknote and the controller controls the transport unit to transport the rejected banknote to the first upper stacking unit or the second upper stacking unit according to a reject factor of the rejected banknote; whereby the controller controls the transport unit to transport the rejected banknote to the first upper stacking unit when a transport error is detected regarding the banknote by a detection unit, and to transport the rejected banknote to the second upper stacking unit when a denomination recognition error or an authentication recognition error is detected regarding the banknote by the recognition unit, or wherein the controller controls the transport unit to transport the rejected banknote to the first upper stacking unit when said transport error is detected for the banknote by said detection unit or when said denomination recognition error is detected regarding the banknote by the recognition unit, and to transport the rejected banknote to the second upper stacking unit when said authentication recognition error is detected regarding the banknote by the recognition unit. a take-in unit that is provided at an upper position of the banknote processing device and takes in the plural banknotes placed on the take-in unit one by one; a recognition unit that recognizes the banknote taken in by the take-in unit; a first upper stacking unit that is provided at an upper position of the banknote processing device; a second upper stacking unit that is provided adjacent to the first upper stacking unit; first to fourth lower stacking units (106a-106d) that are arranged in parallel at lower positions of the banknote processing device and stack banknotes of at least one specific denomination, and each of the first to fourth lower stacking units (106a-106d) includes an opening that opens on a front side of the banknote processing device; a transport unit that transports the banknote taken in by the take-in unit to one of the first and second upper stacking units (102a, 102b) and the first to fourth lower stacking units (106a-106d); and a controller that controls the transport unit to transport the banknote to the first upper stacking unit or the second upper stacking unit when a recognition result obtained by the recognition unit regarding the banknote is a rejected banknote, and to transport the banknote to one of the first to fourth lower stacking units (106a-106d) when the recognition result obtained by the recognition unit regarding the banknote is not a rejected banknote and the controller controls the transport unit to transport the rejected banknote to the first upper stacking unit or the second upper stacking unit according to a reject factor of the rejected banknote;