An exhaust purification system of the internal combustion engine comprising: an upstream side catalyst arranged in an exhaust passage and able to store oxygen; a downstream side catalyst arranged at a downstream side of the upstream side catalyst in the direction of flow of exhaust and able to store oxygen; an air-fuel ratio sensor arranged between the upstream side catalyst and the downstream side catalyst and detecting an air-fuel ratio of outflowing exhaust gas flowing out from the upstream side catalyst; an air-fuel ratio control part configured to control an air-fuel ratio of inflowing exhaust gas flowing into the upstream side catalyst to a target air-fuel ratio; and a temperature calculating part configured to calculate a temperature of the downstream side catalyst, wherein the air-fuel ratio control part is configured to perform first control and second control, in the first control, the air-fuel ratio control part is configured to set the target air-fuel ratio to a first set air-fuel ratio deviating from the stoichiometric air-fuel ratio to a second side at an opposite side from a first side when an air-fuel ratio detected by the air-fuel ratio sensor reaches a judged air-fuel ratio deviating to the first side from the stoichiometric air-fuel ratio, and set the target air-fuel ratio to a second set air-fuel ratio deviating to the first side from the stoichiometric air-fuel ratio when judging that an amount of change of an oxygen storage amount of the upstream side catalyst when the target air-fuel ratio is maintained at an air-fuel ratio deviating to the second side from the stoichiometric air-fuel ratio has reached a reference amount smaller than a maximum oxygen storage amount of the upstream side catalyst, in the second control, the air-fuel ratio control part is configured to set the target air-fuel ratio to a lean set air-fuel ratio leaner than the stoichiometric air-fuel ratio when an air-fuel ratio detected by the air-fuel ratio sensor reaches a rich judged air-fuel ratio richer than the stoichiometric air-fuel ratio, and set the target air-fuel ratio to a rich set air-fuel ratio richer than the stoichiometric air-fuel ratio when the air-fuel ratio detected by the air-fuel ratio sensor reaches a lean judged air-fuel ratio leaner than the stoichiometric air-fuel ratio, and the air-fuel ratio control part is configured to switch the first control to the second control when a temperature of the downstream side catalyst calculated by the temperature calculating part rises to a reference temperature which is equal to or higher than an activation temperature of the downstream side catalyst. an upstream side catalyst arranged in an exhaust passage and able to store oxygen; a downstream side catalyst arranged at a downstream side of the upstream side catalyst in the direction of flow of exhaust and able to store oxygen; an air-fuel ratio sensor arranged between the upstream side catalyst and the downstream side catalyst and detecting an air-fuel ratio of outflowing exhaust gas flowing out from the upstream side catalyst; an air-fuel ratio control part configured to control an air-fuel ratio of inflowing exhaust gas flowing into the upstream side catalyst to a target air-fuel ratio; and a temperature calculating part configured to calculate a temperature of the downstream side catalyst, wherein the air-fuel ratio control part is configured to perform first control and second control, in the first control, the air-fuel ratio control part is configured to set the target air-fuel ratio to a first set air-fuel ratio deviating from the stoichiometric air-fuel ratio to a second side at an opposite side from a first side when an air-fuel ratio detected by the air-fuel ratio sensor reaches a judged air-fuel ratio deviating to the first side from the stoichiometric air-fuel ratio, and set the target air-fuel ratio to a second set air-fuel ratio deviating to the first side from the stoichiometric air-fuel ratio when judging that an amount of change of an oxygen storage amount of the upstream side catalyst when the target air-fuel ratio is maintained at an air-fuel ratio deviating to the second side from the stoichiometric air-fuel ratio has reached a reference amount smaller than a maximum oxygen storage amount of the upstream side catalyst, in the second control, the air-fuel ratio control part is configured to set the target air-fuel ratio to a lean set air-fuel ratio leaner than the stoichiometric air-fuel ratio when an air-fuel ratio detected by the air-fuel ratio sensor reaches a rich judged air-fuel ratio richer than the stoichiometric air-fuel ratio, and set the target air-fuel ratio to a rich set air-fuel ratio richer than the stoichiometric air-fuel ratio when the air-fuel ratio detected by the air-fuel ratio sensor reaches a lean judged air-fuel ratio leaner than the stoichiometric air-fuel ratio, and the air-fuel ratio control part is configured to switch the first control to the second control when a temperature of the downstream side catalyst calculated by the temperature calculating part rises to a reference temperature which is equal to or higher than an activation temperature of the downstream side catalyst.