An optical neural network apparatus comprising: an input layer, a hidden layer, and an output layer sequentially arranged in a traveling direction of light, wherein the output layer comprises an image sensor, the image sensor comprising a plurality of light sensing pixels arranged in two dimensions, and wherein the input layer or the hidden layer comprises at least one passive phase modulator configured to modulate a phase of incident light based on an incident position of the incident light on a two dimensional plane of the at least one passive phase modulator; wherein the at least one passive phase modulator comprises: a transparent substrate; a plurality of phase shifting elements arranged on the substrate, so as to protrude in the form of pillars, wherein each of the plurality of phase shifting elements has a a cylindrical column shape or a radial symmetric polygonal column shape; wherein each phase shifting element from among the plurality of phase shifting elements has a respective diameter depending on a position of the phase shifting element on the substrate; and wherein the respective diameter, parallel to a plane of the substrate, of each phase shifting element from among the plurality of phase shifting elements is based on a respective weight obtained by pre-learning corresponding to the position of the phase shifting element on the substrate, wherein a phase change in light transmitted through a phase shifting element increases as the diameter of the phase shifting element increases. an input layer, a hidden layer, and an output layer sequentially arranged in a traveling direction of light, wherein the output layer comprises an image sensor, the image sensor comprising a plurality of light sensing pixels arranged in two dimensions, and wherein the input layer or the hidden layer comprises at least one passive phase modulator configured to modulate a phase of incident light based on an incident position of the incident light on a two dimensional plane of the at least one passive phase modulator; wherein the at least one passive phase modulator comprises: a transparent substrate; a plurality of phase shifting elements arranged on the substrate, so as to protrude in the form of pillars, wherein each of the plurality of phase shifting elements has a a cylindrical column shape or a radial symmetric polygonal column shape; wherein each phase shifting element from among the plurality of phase shifting elements has a respective diameter depending on a position of the phase shifting element on the substrate; and wherein the respective diameter, parallel to a plane of the substrate, of each phase shifting element from among the plurality of phase shifting elements is based on a respective weight obtained by pre-learning corresponding to the position of the phase shifting element on the substrate, wherein a phase change in light transmitted through a phase shifting element increases as the diameter of the phase shifting element increases. a transparent substrate; a plurality of phase shifting elements arranged on the substrate, so as to protrude in the form of pillars, wherein each of the plurality of phase shifting elements has a a cylindrical column shape or a radial symmetric polygonal column shape; wherein each phase shifting element from among the plurality of phase shifting elements has a respective diameter depending on a position of the phase shifting element on the substrate; and wherein the respective diameter, parallel to a plane of the substrate, of each phase shifting element from among the plurality of phase shifting elements is based on a respective weight obtained by pre-learning corresponding to the position of the phase shifting element on the substrate, wherein a phase change in light transmitted through a phase shifting element increases as the diameter of the phase shifting element increases.