Q:

Neutron discovery, why it couldn't be gamma ray?  or other photon? 

During the discovery of the neutron by J. Chadwick it was observed that something is emitted from beryllium when bombarded by alpha particles that could knock out protons with 5.7 MeV from a paraffin slab.  And they thought it was gamma rays, but later calculated thatthe gamma photon energy would have had to be at least 55 MeV which is far too much energy to be produced from beryllium interacting with an alpha particle, and concluded that it was not a gamma ray. 

A paraffin slab is rich in protons so is there a need of high threshold energy? 
How did they calculate the 55 MeV? 
Can't the photoelectric effect happen there so that only a 5.7 MeV photon is needed? 

A:

This is the illustration of the Chadwick experiment (from Wikipedia):

The key point is that the scattering is close to the forward direction.  So if we draw the diagram for the Compton scattering as:

The angle $\theta$ is small.  This diagram shows the hypothetical incoming gamma ray hitting a proton in the paraffin and scattering it.  The equation for the change in wavelength of the gamma ray (due to energy transfer to the scattered proton) is given by the usual equation for Compton scattering:
$$ \lambda' - \lambda = \frac{h}{m_p\,c}\left(1 - \cos\theta\right) $$
Since $\theta$ is small $\cos\theta\approx 1$ and therefore the the $1-\cos\theta$ term is very small.  This means that the change in wavelength of the gamma photon is small, and therefore that only a small fraction of the gamma photon energy can be transferred to the proton. 
And this is the key point.  Chadwick measured the energy of the emitted protons ($5.5$MeV) and their scattering angle, and used the equation above to calculate what energy the gamma ray had to be in order to transfer $5.5$MeV to the protons.  The result was that that the initial gamma ray had to have an energy of at least 50MeV. But the initil alpha particles didn't have this much energy, so they couldn't possibly create a gamma ray with such a high energy, and that ruled out the presence of gamma rays. 

A:

In 1930 it was discovered that Beryllium, when bombarded by alpha particles, emitted a very energetic stream of radiation.  This stream was originally thought to be gamma radiation.   However, further investigations into the properties of the radiation revealed contradictory results.  Like gamma rays, these rays were extremely penetrating and since they were not deflected upon passing through a magnetic field, neutral.  However, unlike gamma rays, these rays did not discharge charged electroscopes (the photoelectric effect).  Irene Curie and her husband discovered that when a beam of this radiation hit a substance rich in protons, for example paraffin, protons were knocked loose which could be easily detected by a Geiger counter. 
In 1932, Chadwick proposed that this particle was Rutherford's neutron.  In 1935, he was awarded the Nobel Prize for his discovery.  Using kinematics, Chadwick was able to determine the velocity of the protons.  Then through conservation of momentum techniques, he was able to determine that the mass of the neutral radiation was almost exactly the same as that of a proton.  This is Chadwick's equation:

italics mine. 
The answers to your questions:

paraffin slab is rich in protons so is there a need of high threshold energy? 

The need for energy comes from the measured energy of the scattered by "X" particle

How they calculated the 55 Mev? 

they measured the energy of the scattered proton. 

Can't photo electric effect happen there so that only 5.7 Mev photon is needed? 

The photoelectric effect is at the atom level, not the nucleus level and the energies are of order keV not MeV. In addition (italics in quote) the "X" did not interact electromagnetically.