A processor-implemented method for real-time hand detection in a first person view, the method comprising: receiving a RGB image of a scene captured in the FPV as an input by a Convolutional Neural Network, via one or more hardware processors, characterized in that, the CNN comprising eleven learned layers having nine convolutional layers followed by two fully connected layers; extracting image features from the RGB image by the nine convolutional layers of the CNN, via the one or more hardware processors; and predicting, based on the image features, a probability of detection of the hand, a normalized centroid of the hand and normalized dimensions of a bounding box around the hand in the RGB image, via the one or more hardware processors, by the fully connected layer of the CNN, wherein coordinates of the bounding box are absolute image coordinates, wherein the CNN is pre-trained based on a loss function represented by the equation: L ( ∧ y ( ∧ c x, ∧ c y, ∧ h, ∧ w), y ( p, c x, c y, h, w)) = α L c o n f i d e n c e + p * ( β L c e n t r o i d + γ L b o u n d i n g b o x) where, L confidence = -p log( ^ p) - (1-p) log (1- ^ p) represents confidence loss, L centroid = ( c x - ^ c x) 2 + ( c y - ^ c y) 2 represents centroid loss, L boundingbox = h − ∧ h h + w − ∧ w w represents bounding box dimension loss, where, p is the probability of hand in the FPV, the probability is 0 if the hand is absent in the FPV, and the probability is 1 if the hand is present in the FPV, ( c x; c y), h, w is ground truth of hand's bounding box's centroid, height and width, respectively, p̂ is the predicted probability in the field of view, ( ^ c x, ^ c y), ^ h, ^ w is predicted hand bounding box's centroid, height and width, α, β, γ are scalars that weigh the confidence loss, the centroid loss, and the bounding box dimension loss respectively, wherein the loss function enables reflection of small deviations in a large bounding box that matter less than small deviations in a small bounding box as magnitude of loss is given more weightage in the small bounding box compared to the large bounding box in the hand detection. receiving a RGB image of a scene captured in the FPV as an input by a Convolutional Neural Network, via one or more hardware processors, characterized in that, the CNN comprising eleven learned layers having nine convolutional layers followed by two fully connected layers; extracting image features from the RGB image by the nine convolutional layers of the CNN, via the one or more hardware processors; and predicting, based on the image features, a probability of detection of the hand, a normalized centroid of the hand and normalized dimensions of a bounding box around the hand in the RGB image, via the one or more hardware processors, by the fully connected layer of the CNN, wherein coordinates of the bounding box are absolute image coordinates, wherein the CNN is pre-trained based on a loss function represented by the equation: L ( ∧ y ( ∧ c x, ∧ c y, ∧ h, ∧ w), y ( p, c x, c y, h, w)) = α L c o n f i d e n c e + p * ( β L c e n t r o i d + γ L b o u n d i n g b o x) where, L confidence = -p log( ^ p) - (1-p) log (1- ^ p) represents confidence loss, L centroid = ( c x - ^ c x) 2 + ( c y - ^ c y) 2 represents centroid loss, L boundingbox = h − ∧ h h + w − ∧ w w represents bounding box dimension loss, where, p is the probability of hand in the FPV, the probability is 0 if the hand is absent in the FPV, and the probability is 1 if the hand is present in the FPV, ( c x; c y), h, w is ground truth of hand's bounding box's centroid, height and width, respectively, p̂ is the predicted probability in the field of view, ( ^ c x, ^ c y), ^ h, ^ w is predicted hand bounding box's centroid, height and width, α, β, γ are scalars that weigh the confidence loss, the centroid loss, and the bounding box dimension loss respectively, wherein the loss function enables reflection of small deviations in a large bounding box that matter less than small deviations in a small bounding box as magnitude of loss is given more weightage in the small bounding box compared to the large bounding box in the hand detection. L confidence = -p log( ^ p) - (1-p) log (1- ^ p) represents confidence loss, L centroid = ( c x - ^ c x) 2 + ( c y - ^ c y) 2 represents centroid loss, L boundingbox = h − ∧ h h + w − ∧ w w represents bounding box dimension loss, where, p is the probability of hand in the FPV, the probability is 0 if the hand is absent in the FPV, and the probability is 1 if the hand is present in the FPV, ( c x; c y), h, w is ground truth of hand's bounding box's centroid, height and width, respectively, p̂ is the predicted probability in the field of view, ( ^ c x, ^ c y), ^ h, ^ w is predicted hand bounding box's centroid, height and width, α, β, γ are scalars that weigh the confidence loss, the centroid loss, and the bounding box dimension loss respectively, wherein the loss function enables reflection of small deviations in a large bounding box that matter less than small deviations in a small bounding box as magnitude of loss is given more weightage in the small bounding box compared to the large bounding box in the hand detection.