Package: elegant-heap-queue
Elegant Heap QueueAt last a better and more elegant Heap Queue library for Python! We developed this library as a result of our interactions with established Python PriorityQueue and heapq libraries through Carnegie Mellon computing courses and Leetcode grinding. Explicitly, we noticed that the Python standard libraries, while useful and implementing correct heap behavior, do not allow for expressive code. They have limitations which lead to potentially opaque and difficult to maintain code, using several "hacks" to achieve what is normal in other languages like Java. To understand better our pain points, consider a classic Leetcode question--Median Finder. Below shows an implementation that uses the heapq Python standard library...classMedianFinderStandard:def__init__(self):# smaller half of data. maxHeap[0] stores the biggest data in smaller halfself.maxHeap=[]# bigger half of data. minHeap[0] stores the smallest data in bigger halfself.minHeap=[]defaddNum(self,num:int)->None:iflen(self.maxHeap)>len(self.minHeap):x=heapq.heappushpop(self.maxHeap,-num)heapq.heappush(self.minHeap,-x)else:y=heapq.heappushpop(self.minHeap,num)heapq.heappush(self.maxHeap,-y)deffindMedian(self)->float:# if number of element is odd, median is biggest element in maxHeapif(len(self.maxHeap)+len(self.minHeap))%2!=0:return-(self.maxHeap[0])# if number of element is even, median is the mean of two heaps top elemsreturn(-self.maxHeap[0]+self.minHeap[0])/2Compare this code to a re-implementation using our library...classMedianFinderElegant:def__init__(self):self.maxHeap=HeapQueue(heap_type=HeapType.MAX_HEAP)self.minHeap=HeapQueue(heap_type=HeapType.MIN_HEAP)defaddNum(self,num:int)->None:iflen(self.maxHeap)>len(self.minHeap):self.maxHeap.push(num)x=self.maxHeap.pop()self.minHeap.push(x)else:self.minHeap.push(num)y=self.minHeap.pop()self.maxHeap.push(y)# negations no moredeffindMedian(self)->float:# if number of element is odd, median is biggest element in maxHeapif(len(self.maxHeap)+len(self.minHeap))%2!=0:return(self.maxHeap.peek())# if number of element is even, median is the mean of two heaps top elemsreturn(self.maxHeap.peek()+self.minHeap.peek())/2Hopefully, you will notice that latter is significantly more expressive. The code is cleaner and it tells you exactly what it is doing, there is no need for negations or understanding of how the library works under-the-hood to understand the code: what you see is what you get. The Python PQ libraries also get worse when we start to involve classes, a problem since Python is used often in an Object-Oriented-Programming manner in real applications. Below are two different Java code snippets from Pinterest's memq project and Hadoop hdds using PQs with classes:Map<String,PriorityQueue<Broker>>rackBrokerCapacityMap=newHashMap<>();for(Brokerbroker:brokerList){Integercapacity=throughputMap.getOrDefault(broker.getInstanceType(),DEFAULT_CAPACITY);if(broker.getTotalNetworkCapacity()==0){broker.setTotalNetworkCapacity(capacity);}PriorityQueue<Broker>priorityQueue=rackBrokerCapacityMap.computeIfAbsent(broker.getLocality(),k->newPriorityQueue<>((o1,o2)->Integer.compare(o2.getAvailableCapacity(),o1.getAvailableCapacity())));priorityQueue.add(broker);}FindSourceGreedy(NodeManagernodeManager){sizeLeavingNode=newHashMap<>();potentialSources=newPriorityQueue<>((a,b)->{doublecurrentUsageOfA=a.calculateUtilization(-sizeLeavingNode.get(a.getDatanodeDetails()));doublecurrentUsageOfB=b.calculateUtilization(-sizeLeavingNode.get(b.getDatanodeDetails()));//in descending orderintret=Double.compare(currentUsageOfB,currentUsageOfA);if(ret!=0){returnret;}UUIDuuidA=a.getDatanodeDetails().getUuid();UUIDuuidB=b.getDatanodeDetails().getUuid();returnuuidA.compareTo(uuidB);});this.nodeManager=nodeManager;}In standard Python, you cannotintegrate the logic to get "priority" for the HeapQueue into the data structure itself! It has to be done at eachheappush(...)call using a tuple hack:(item_priority, item). Alternatively, it can be achieved by overriding a classes__lt__()method; however, these hacks all lead back to our initial problem with lack of expression. These hacks all require knowledge of how the libraries (heapq, queue) are implemented in order to "hack" together solutions. The result is code which is difficult to maintain, easy to generate subtle bugs, and hard to read. With our library, this feature is enabled which allows a HeapQueue to encapsulate the logic for generating its items priority via a "key function". In essence, you define it once at creation time and never worry about it again. The Java excerpts could be re-implemented in our library like so...rack_broker_capacity_dict={}forbrokerinbrokerList:capacity=throughput_dict.get(broker.get_instance_type(),DEFAULT_CAPACITY)ifbroker.get_total_network_capacity()==0:broker.set_total_network_capacity(capacity)pq=rack_broker_capacity_dict.get(broker.get_locality(),HeapQueue(heap_type=HeapType.MIN_HEAP,key=lambdao:o.get_available_capacity()))'''Push and pop items from pq object with Heap guarantees using key for"priority" and heap_type for ordering priority'''deffind_source_greedy(node_manager):size_leaving_node={}potential_sources=HeapQueue(heap_type=HeapType.MIN_HEAP,key=lambdaa:(# tuple key for fallback priorities if equal on firsta.calculate_utilization(-size_leaving_node.get(a.get_datanode_details())),a.get_datanode_details().get_uuid()))self.node_manager=node_managerSimilar real world examples where this would be useful can involve load balancing, scheduling, encoding, algorithms, and more, where classes that encapsulate lots of knowledge are used and an objects "priority" is not always a simple calculation. Inspirations for the library stem from Java and C++ implementations as well as use of the Python platform. We hope that you find this library useful in your future endeavors and want to give a shout-out to our ProfessorCharlie Garrod. For more information on the library see the sections below.InstallationWith a standard Python installation, to install do:pip install elegant-heap-queuethen import like so:fromelegant_heap_queueimport*# HeapQueue and HeapType class importedIt's also possible to use the library files uploaded to the PyPIwebsitedirectly in your projects.Getting Started with Elegant Priority QueueConstructor ArgumentsThe most basic way to instantiate a HeapQueue,heap = HeapQueue(). However, there are many different
arguments that help elevate the usefulness of this library:Argument NameDescriptionExampleitemsIterable of initial items to push to the HeapQueueheap = HeapQueue(items=[1, 2, 3])heap_typeDefines the ordering of the items within the HeapQueue. Default value MIN_HEAPheap = HeapQueue(HeapQueue(heap_type=HeapType.MAX_HEAP))keyKey function to define the value which an item's "priority" should be generated from. Generally, used when the HeapQueue's type is a custom classheap = HeapQueue(key=lambda x: x.val)Core FunctionsFunctionDescriptionExamplepeekReturns (but does not remove) the highest priority itemheap.peek()pushPushes theitemonto the HeapQueue, while maintaining heap invariantheap.push(1)push_allTakes an iterable which will add all elements to heapheap.push([1, 2, 3])popReturns and removes the highest priority itemheap.pop()pop_kReturns and removes thekhighest priority itemsheap.pop_k(2)as_sorted_listReturns all items in the HeapQueue as a sorted list based on priorityfor item in heap.as_sorted_list(): ...LicenseMITAuthorsAlexDavidTylerShou-San