Question: I am unsure how to extra the 'Settle' column for each state (New South Wales, Victoria, Queensland, South Australia) from this website: https://www.asxenergy.com.au/futures_au It seems the numerical data isn't showing. My starting code is: from bs4 import BeautifulSoup from urllib.request import urlopen url = "https://www.asxenergy.com.au/futures_au" page = urlopen(url) html = page.read().decode("utf-8") soup = BeautifulSoup(html, "html.parser") print(soup.get_text()) 

Answer: As stated, the data comes from asxenergy.com.au/futures_au/dataset Use pandas to get the <table> tags. It's not ideally structured, so need a bit of processing here. import pandas as pd url = 'https://www.asxenergy.com.au/futures_au/dataset' dfs = pd.read_html(url) states = dfs[0].iloc[0].to_list() table_names = list(dfs[1].columns[0]) data_dfs = dfs[2:] output = {} for count, state in enumerate(states): start_idx = count * 6 end_idx = start_idx + 6 set_idx = list(range(start_idx, end_idx)) temp_table_names = [table_names[i] for i in set_idx] temp_data_dfs = [data_dfs[i] for i in set_idx] data_dict = dict(zip(temp_table_names, temp_data_dfs)) output[state] = data_dict  Output: for state, tables in output.items(): for table_name, df in tables.items(): print(state, '-', table_name) print(df, '\n') New South Wales - BMth BMth Bid Ask Last +/- Vol Settle 0 Jul24 - - - - - 130.75 1 Aug24 - - - - - 117.74 2 Sep24 - - - - - 97.43 3 Oct24 - - - - - 110.00 New South Wales - BQtr BQtr Bid Ask Last +/- Vol Settle 0 Q324 113.50 115.50 115.50 - 40 115.50 1 Q424 109.50 110.00 110.00 - 16 110.00 2 Q125 126.10 126.70 126.10 - 16 126.10 3 Q225 130.15 130.50 130.15 - 21 130.15 4 Q325 125.00 126.85 126.18 - 23 126.18 New South Wales - BStr BStr Bid Ask Last +/- Vol Settle 0 FY25 120.29 121.00 - - - 120.38 1 CY25 121.00 122.50 122.00 - 14 122.00 2 FY26 120.00 122.00 121.00 -0.19 4 121.19 3 CY26 120.50 121.25 121.50 +0.69 7 120.81 4 FY27 120.25 121.00 - - - 120.43 New South Wales - Caps Caps Bid Ask Last +/- Vol Settle 0 Q324 10.00 11.00 10.00 - 17 10.00 1 Q424 20.25 21.50 20.50 +0.25 10 20.25 2 Q125 41.00 43.00 - - - 41.46 3 Q225 25.50 27.00 - - - 26.66 4 Q325 25.00 - - - - 26.54 New South Wales - CapStr CapStr Bid Ask Last +/- Vol Settle 0 FY25 - - - - - 24.49 1 CY25 - - - - - 28.96 2 FY26 29.40 - - - - 29.40 3 CY26 28.50 30.00 29.00 -0.35 1 29.35 4 FY27 29.00 30.00 - - - 29.93 New South Wales - PQtr PQtr Bid Ask Last +/- Vol Settle 0 Q324 116.00 160.00 - - - 160.00 1 Q424 125.00 160.00 - - - 145.00 2 Q125 - - - - - 170.73 3 Q225 - - - - - 150.00 4 Q325 - - - - - 140.00 Victoria - PkStr PkStr Bid Ask Last +/- Vol Settle 0 FY25 - - - - - 156.47 1 CY25 - - - - - 144.42 2 FY26 - - - - - 147.97 3 CY26 - - - - - 160.00 4 FY27 - - - - - 148.94 Victoria - BMth BMth Bid Ask Last +/- Vol Settle 0 Jul24 - - - - - 142.58 1 Aug24 - - - - - 108.21 2 Sep24 - 80.00 - - - 71.28 3 Oct24 - - - - - 58.95 Victoria - BQtr BQtr Bid Ask Last +/- Vol Settle 0 Q324 107.75 108.50 107.75 - 18 107.75 1 Q424 58.50 58.95 59.00 +0.05 14 58.95 2 Q125 73.40 74.00 73.50 - 13 73.50 3 Q225 93.00 93.80 93.80 - 10 93.80 4 Q325 88.00 89.25 88.50 - 5 88.50 Victoria - BStr BStr Bid Ask Last +/- Vol Settle 0 FY25 - - - - - 83.53 1 CY25 76.25 76.85 76.65 +0.29 2 76.36 2 FY26 70.00 73.30 - - - 72.22 3 CY26 65.50 70.24 69.80 +0.25 8 69.55 4 FY27 - - - - - 70.17 Victoria - Caps Caps Bid Ask Last +/- Vol Settle 0 Q324 6.75 7.5 8.15 +0.65 18 7.5 1 Q424 5.00 7.0 - - - 6.3 2 Q125 27.75 29.0 28.00 - 20 28.0 3 Q225 8.25 11.5 - - - 10.0 4 Q325 10.50 11.5 - - - 10.5 Victoria - CapStr CapStr Bid Ask Last +/- Vol Settle 0 FY25 - - - - - 12.88 1 CY25 13.00 15.25 - - - 14.25 2 FY26 13.85 15.25 - - - 14.49 3 CY26 14.50 16.00 - - - 14.81 4 FY27 15.40 - - - - 15.42 Queensland - PQtr PQtr Bid Ask Last +/- Vol Settle 0 Q324 111.11 - - - - 140.0 1 Q424 - - - - - 64.5 2 Q125 - - - - - 103.0 3 Q225 - - - - - 120.0 4 Q325 - - - - - 119.0 Queensland - PkStr PkStr Bid Ask Last +/- Vol Settle 0 FY25 - - - - - 107.07 1 CY25 - - - - - 102.24 2 FY26 - - - - - 77.04 3 CY26 - - - - - 60.00 4 FY27 - - - - - 60.00 Queensland - BMth BMth Bid Ask Last +/- Vol Settle 0 Jul24 - - - - - 108.07 1 Aug24 - - - - - 89.86 2 Sep24 - - - - - 89.86 3 Oct24 - - - - - 97.00 Queensland - BQtr BQtr Bid Ask Last +/- Vol Settle 0 Q324 96.00 101.75 - - - 96.00 1 Q424 96.75 97.50 97.00 - 12 97.00 2 Q125 133.00 133.50 133.50 - 18 133.50 3 Q225 99.80 100.45 99.80 - 31 99.80 4 Q325 95.00 96.75 96.01 - 15 96.01 Queensland - BStr BStr Bid Ask Last +/- Vol Settle 0 FY25 106.50 109.75 - - - 106.45 1 CY25 104.00 104.50 104.00 -0.22 7 104.22 2 FY26 99.50 100.00 99.75 -0.12 4 99.87 3 CY26 97.00 98.00 97.40 -0.02 2 97.42 4 FY27 - 94.70 94.50 +0.11 2 94.39 Queensland - Caps Caps Bid Ask Last +/- Vol Settle 0 Q324 7.00 9.00 8.25 - 15 8.25 1 Q424 17.25 18.90 19.00 +0.10 1 18.90 2 Q125 42.00 45.25 45.00 - 1 45.00 3 Q225 - 17.00 - - - 17.00 4 Q325 - - - - - 16.84 South Australia - CapStr CapStr Bid Ask Last +/- Vol Settle 0 FY25 - - - - - 22.18 1 CY25 - 23.85 - - - 23.74 2 FY26 - 23.30 - - - 23.19 3 CY26 21.50 23.50 22.99 - 10 22.99 4 FY27 - - - - - 22.78 South Australia - PQtr PQtr Bid Ask Last +/- Vol Settle 0 Q324 95.0 - - - - 95.0 1 Q424 100.0 - - - - 100.0 2 Q125 160.0 - - - - 160.0 3 Q225 100.0 - - - - 100.0 4 Q325 90.0 - - - - 90.0 South Australia - PkStr PkStr Bid Ask Last +/- Vol Settle 0 FY25 - - - - - 113.53 1 CY25 - - - - - 107.21 2 FY26 - - - - - 69.79 3 CY26 - - - - - 54.00 4 FY27 - - - - - 54.00 South Australia - BMth BMth Bid Ask Last +/- Vol Settle 0 Jul24 - - - - - 129.78 1 Aug24 - - - - - 129.78 2 Sep24 - - - - - 129.78 3 Oct24 - - - - - 67.32 South Australia - BQtr BQtr Bid Ask Last +/- Vol Settle 0 Q324 - 154.75 - - 2 129.78 1 Q424 - - - - 2 67.32 2 Q125 108.50 120.00 - - 2 117.08 3 Q225 - 123.50 - - 2 116.12 4 Q325 - - - - - 119.00 South Australia - BStr BStr Bid Ask Last +/- Vol Settle 0 FY25 - - 107.50 - 2 107.50 1 CY25 - - - - - 105.90 2 FY26 - 108.00 - - - 104.38 3 CY26 - - - - - 103.45 4 FY27 100.00 105.00 - - - 103.14