Question: I have a list of dict that follow a consistent structure where each dict has a list of integers. However, I need to make sure each dict has a bytesize (when converted to a JSON string) less than a specified threshold. If the dict exceeds that bytesize threshold, I need to chunk that dict's integer list. Attempt:  import json payload: list[dict] = [ {"data1": [1,2,3,4]}, {"data2": [8,9,10]}, {"data3": [1,2,3,4,5,6,7]} ] # Max size in bytes we can allow. This is static and a hard limit that is not variable. MAX_SIZE: int = 25 def check_and_chunk(arr: list): def check_size_bytes(item): return True if len(json.dumps(item).encode("utf-8")) > MAX_SIZE else False def chunk(item, num_chunks: int=2): for i in range(0, len(item), num_chunks): yield item[i:i+num_chunks] # First check if the entire payload is smaller than the MAX_SIZE if not check_size_bytes(arr): return arr # Lets find the items that are small and items that are too big, respectively small, big = [], [] # Find the indices in the payload that are too big big_idx: list = [i for i, j in enumerate(list(map(check_size_bytes, arr))) if j] # Append these items respectively to their proper lists item_append = (small.append, big.append) for i, item in enumerate(arr): item_append[i in set(big_idx)](item) # Modify the big items until they are small enough to be moved to the small_items list for i in big: print(i) # This is where I am unsure of how best to proceed. I'd like to essentially split the big dictionaries in the 'big' list such that it is small enough where each element is in the 'small' result.  Example of a possible desired result: payload: list[dict] = [ {"data1": [1,2,3,4]}, {"data2": [8,9,10]}, {"data3": [1,2,3,4]}, {"data3": [5,6,7]} ] 

Answer: As discussed in the comments, a simple approach to this task is to recursively split the input list until the output dict meets the size requirement. This will give more evenly sized lists in the output, but may result in more dicts than are absolutely necessary (and would be produced by one of the length accumulation approaches). import json def split_list_dict(dl, limit): def split_dict_list(dd, limit): def json_len(ll): return sum(map(len, map(str, ll))) + 2 * len(ll) # 2 * len(ll) allows for [] and , ll = next(iter(dd.values())) key = next(iter(dd.keys())) dict_jsonlen = len(json.dumps(dd)) if dict_jsonlen <= limit: yield dd return list_jsonlen = json_len(ll) keylen = dict_jsonlen - list_jsonlen split_point = len(ll) // 2 yield from split_dict_list({ key : ll[:split_point] }, limit) yield from split_dict_list({ key : ll[split_point:] }, limit) for dd in dl: yield from split_dict_list(dd, limit) MAX_SIZE: int = 25 payload: list[dict] = [ {"data1": [1, 2, 3, 4]}, {"long_data_name": [1, 2, 3, 4]}, {"data3": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]}, {"data4": [100, 200, -1, -10, 200, 300, 12, 13]}, ] print(list(split_list_dict(payload, MAX_SIZE)))  Output: [ {'data1': [1, 2, 3, 4]}, {'long_data_name': [1]}, {'long_data_name': [2]}, {'long_data_name': [3]}, {'long_data_name': [4]}, {'data3': [1, 2, 3]}, {'data3': [4, 5, 6]}, {'data3': [7, 8, 9]}, {'data3': [10, 11, 12]}, {'data4': [100, 200]}, {'data4': [-1, -10]}, {'data4': [200, 300]}, {'data4': [12, 13]} ]