Repository: jsvine/pdfplumber

Issue #265: Fewer horizontal lines when using text strategy
Created: 2020-09-01T19:46:49Z
Describe the bug
In d224202, the logic for finding horizontal and vertical lines connecting n number of words was simplified. When finding the horizontal lines, the logic was updated to keep the "top" of all line rects and the bottom of only the last line rect. This is causing problems with table detection as the final number of horizontal lines has reduced and when the gap between 2 rows is big, it can provide inconsistent results when used together with snap_tolerance.
The height of the line is also not in sync with the height of text it possesses.
Code to reproduce the problem
import pdfplumber

pdf = pdfplumber.open("issue-67-example.pdf")
p = pdf.pages[20]
ts = {"vertical_strategy": "lines", "horizontal_strategy": "text"}
im = p.to_image(resolution=150)
im.reset().debug_tablefinder(ts)
im.save("image.png", format="PNG")

tables = p.extract_tables(table_settings=ts)
for table in tables:
    for row in table:
        print(row)
PDF file
The PDF file can be found here.
Expected behavior
On versions before v0.5.23
Last row of the table: ['金', '', '']

Actual behavior
On v0.5.23
Last row of the table: ['支付其他与投资活动有关的现', '', '']

Environment

pdfplumber version: 0.5.23
Python version: 3.8.2
OS: Linux

Additional context
Causes trouble when the vertical spacing between 2 words is big.
To some context, the change makes sense as a horizontal row of text is sandwiched between 2 lines and there are no consecutive empty lines (as can be found in the Expected Behavior's screenshot). What could be debated is where to put the line between 2 rows of text? Should it be in the middle (purple line)? Top of the bottom row (green line)? Bottom of the top row (orange line)? The current implementation has picked the top of the bottom row.

I would prefer that it is reverted to the older approach in which both the top of the bottom row and bottom of the top row were kept and leave it up to the user to filter since there is no one-filter-suits-all. What are your thoughts @jsvine ?
TODO (from: @samkit-jain ): I looked at the horizontal edges but perhaps it affected vertical edges as well and I should test that out as well.

Duplicate issue #268: Use the extract_table() method to parse out such a table
Created: 2020-09-09T03:51:58Z
When I parse the pdf, I use the extract_table() method, whether the parameter information is passed ({ "vertical_strategy": "lines", "horizontal_strategy": "lines", }) Neither lines nor  text can read the complete form information
The pdf in question is as follows:
error_dpf_3.pdf
Using lines, the table cannot be detected, using text, the parsed result is wrong：
平安银行股份有限公司
资产负债表
2019年12月31日(除特别注明外，金额单位均为人民币百万元)
Can you help me see how the pdf of the table in this situation can be parsed?

Closing comment (2020-09-09T06:12:42Z): Hi @CuteyBoy Could you confirm what version of pdfplumber are you using?
Running on 0.5.23, and using the following code
import pdfplumber

pdf = pdfplumber.open("file.pdf")
p = pdf.pages[0]

ts = {
    "vertical_strategy": "text",
    "horizontal_strategy": "text",
}

im = p.to_image(resolution=150)
im.reset().debug_tablefinder(ts)
im.save("out.png", format="PNG")

tables = p.extract_tables(table_settings=ts)

for table in tables:
    for row in table:
        print(row)
I am getting the following response which appears to be almost correct
['平安银行股份有限公司', '', '', '']
['资产负债表', '', '', '']
['2019年12月31日', '', '', '']
['(除特别注明外，金额单位均为人民币百万元)', '', '', '']
['', '附注三', '2019年12月31日', '2018年12月31日']
['资产', '', '', '']
['现金及存放中央银行款项', '1', '252,230', '278,528']
['存放同业款项', '2', '85,684', '85,098']
['贵金属', '', '51,191', '56,835']
['拆出资金', '3', '79,369', '72,934']
['衍生金融资产', '4', '18,500', '21,460']
['买入返售金融资产', '5', '62,216', '36,985']
['发放贷款和垫款', '6', '2,259,349', '1,949,757']
['金融投资：', '', '', '']
['交易性金融资产', '7', '206,682', '148,768']
['债权投资', '8', '656,290', '629,366']
['其他债权投资', '9', '182,264', '70,664']
['其他权益工具投资', '10', '1,844', '1,519']
['投资性房地产', '11', '247', '194']
['固定资产', '12', '11,092', '10,899']
['使用权资产', '13', '7,517', '-']
['无形资产', '14', '4,361', '4,771']
['商誉', '15', '7,568', '7,568']
['递延所得税资产', '16', '34,725', '29,468']
['其他资产', '17', '17,941', '13,778']


The bottom line is missing and that is related to #265
As a workaround, you can crop the bottom portion of the page (p = p.crop((0, 0, p.width, p.height-100))) and then rerun which would give you the following response:
['平安银行股份有限公司', '', '', '']
['资产负债表', '', '', '']
['2019年12月31日', '', '', '']
['(除特别注明外，金额单位均为人民币百万元)', '', '', '']
['', '附注三', '2019年12月31日', '2018年12月31日']
['资产', '', '', '']
['现金及存放中央银行款项', '1', '252,230', '278,528']
['存放同业款项', '2', '85,684', '85,098']
['贵金属', '', '51,191', '56,835']
['拆出资金', '3', '79,369', '72,934']
['衍生金融资产', '4', '18,500', '21,460']
['买入返售金融资产', '5', '62,216', '36,985']
['发放贷款和垫款', '6', '2,259,349', '1,949,757']
['金融投资：', '', '', '']
['交易性金融资产', '7', '206,682', '148,768']
['债权投资', '8', '656,290', '629,366']
['其他债权投资', '9', '182,264', '70,664']
['其他权益工具投资', '10', '1,844', '1,519']
['投资性房地产', '11', '247', '194']
['固定资产', '12', '11,092', '10,899']
['使用权资产', '13', '7,517', '-']
['无形资产', '14', '4,361', '4,771']
['商誉', '15', '7,568', '7,568']
['递延所得税资产', '16', '34,725', '29,468']
['其他资产', '17', '17,941', '13,778']
['资产总计', '', '3,939,070', '3,418,592']


Does this resolve your issue?