A method of identifying the direction of characters in an image block, comprising the steps of: performing optical character recognition processing on the image block by assuming various directions as assumed character directions, respectively, to obtain sub image blocks, recognized characters corresponding to the sub image blocks and correctness measures of the recognized characters in each of the assumed character directions; in sub image blocks in the assumed character directions with a 180° mutual relation therebetween, searching for a matching pair of the sub image blocks, wherein the matching pair is two sets of sub image blocks in the assumed character directions with a 180° mutual relation therebetween, wherein any one of the two sets of sub image blocks of the matching pair, after being rotated by 180° along with the entire image block, will superimpose upon the other set of the two sets of sub image blocks of the matching pair and the two sets of sub image blocks satisfying the above condition include a minimum number of sub image blocks determining whether the numbers of sub image blocks in the two assumed character directions in the minimum matching pair are identical and; when the numbers of sub image blocks in the assumed character directions in the matching pair are not identical, adjusting: (i) the correctness measures of the sub image blocks in the searched matching pair so that the correctness measures in the set of sub image blocks with the smaller number of sub image blocks are increased or the correctness measures in the set of sub image blocks with the larger number of sub image blocks are decreased, or (ii) the number of the sub image blocks in the searched matching pair so that the number of sub image blocks in the two sets of sub image blocks are identical; calculating an accumulative correctness measure in each of the assumed character directions based on the potentially adjusted correctness measures or number of sub image blocks; and identifying the direction of the characters in the image block according to the accumulative correctness measures. performing optical character recognition processing on the image block by assuming various directions as assumed character directions, respectively, to obtain sub image blocks, recognized characters corresponding to the sub image blocks and correctness measures of the recognized characters in each of the assumed character directions; in sub image blocks in the assumed character directions with a 180° mutual relation therebetween, searching for a matching pair of the sub image blocks, wherein the matching pair is two sets of sub image blocks in the assumed character directions with a 180° mutual relation therebetween, wherein any one of the two sets of sub image blocks of the matching pair, after being rotated by 180° along with the entire image block, will superimpose upon the other set of the two sets of sub image blocks of the matching pair and the two sets of sub image blocks satisfying the above condition include a minimum number of sub image blocks determining whether the numbers of sub image blocks in the two assumed character directions in the minimum matching pair are identical and; when the numbers of sub image blocks in the assumed character directions in the matching pair are not identical, adjusting: (i) the correctness measures of the sub image blocks in the searched matching pair so that the correctness measures in the set of sub image blocks with the smaller number of sub image blocks are increased or the correctness measures in the set of sub image blocks with the larger number of sub image blocks are decreased, or (ii) the number of the sub image blocks in the searched matching pair so that the number of sub image blocks in the two sets of sub image blocks are identical; (i) the correctness measures of the sub image blocks in the searched matching pair so that the correctness measures in the set of sub image blocks with the smaller number of sub image blocks are increased or the correctness measures in the set of sub image blocks with the larger number of sub image blocks are decreased, or (ii) the number of the sub image blocks in the searched matching pair so that the number of sub image blocks in the two sets of sub image blocks are identical; calculating an accumulative correctness measure in each of the assumed character directions based on the potentially adjusted correctness measures or number of sub image blocks; and identifying the direction of the characters in the image block according to the accumulative correctness measures.