Package: lyne
LynePython processing pipelinesUsing generators, operations and proxy objects for convenient construction of pipelines.Lyne is a way to easily write a processing pipeline.
Combine operations, add conditions and process any stream of input data.Lyne uses a few classes to make the definitions easy to write and understand.Item, StreamAnItemis a convenience class inheriting a dictionary. You can access the values by using the keys as attributes.item.keywill simply returnitem['key']. Another convenience is that it will returnNonefor missing keys. A special field isskip; this value will define if the item should be skipped from further processing. It can be any truthy value, including a textual reason for skipping that can be used for later filtering or debugging.AStreamis simply a wrapper around an iterator. It's only used to differentiate between iterable arguments and actual streams of data that need to be processed.OperationAnOperationis a wrapper around a function that is performed on anItem. TheOperationitself accepts aStreamand will output aStream. (The first in a pipeline can also accept anItem)The arguments to anOperationwill be placeholders. Only during processing will these placeholders be filled with the actual data needed for the wrapped function.ProxyAProxyis a placeholder that accepts all kinds of (simple) operations and aplies them at a later time on an actual object. For example:placeholder = ItemProxy().value * 10 + 5will remember the operations applied to it. Later you can callobj.value = 3; Proxy.apply(placeholder, obj)and it will return 35. (First it retrieves the value of obj.value, then multiplies by 10 and lastly adds 5 to it.)Proxyobjects are used for the placeholders ofOperationobjects; both as arguments as well as outputs. It makes it easy to writeopen_image(I.path) >> I.image. When defining thisOperationit will store the placeholders. When processing the pipeline, for each item that is being processed the placeholder will be replaced by the actual value of the item.>>defines the output location, so in this example an image will be opened using item.path, and saved to item.image.RelativeValueRelativeValueare used in places where you want to specify a value in relation to another. Want to shrink an image to 50%? Threshold a grayscale image at 10%? Rescale an array where the values between 10% and 50% map to 0 and 255?RelativeValuewill make this easy.Note that the function must be aware of these relative values and apply min-max to get an absolute value.
The package modules take advantage of this where possible.ModulesfsThis is the basic filesystem module. It contains simple path manipulation.utilAssistance module that contains basic functions.cvOpenCV based image module, used for image calculations, manipulation, detection, etc.clipCLIPText based module. Used for image captioning, segmentation and other cool CLIP stuff.depthMiDaS based depth module. Used to generate depth maps.Convenience functionsThe core module contains a number of convenience function to make  writing pipelines easier and more readable.Op, CondOpis a convenience object for creating a newOperation. If you writeop_print = Op[print]it will wrap the print function into anOperation. Afterwards you can use thisOperationby calling it with placeholders like so:op_print(I.value_to_print, ':', I.other_value * 10).Or you can combine them all in one line:Op[print](I.value)(The square brackets around the function instead of parentheses have been deliberately chosen for readability).The right shift operator (>>) is used to conveniently define the output mapping.Op[max](I.val_a, I.val_b) >> I.resultwill calculate the max of the two input values and store the result into the item'sresultattribute.Condis a simpleOperationthat will store the second argument initem.skipif the first argument is true. It makes it easy to sayCond(I.value <= 0, 'value too low'); if item.value is zero or lower, the item will be skipped from future processing. The reason will be 'value too low'.S, I, OS, I and O are used to define Item, Stream and Output proxies, respectively.If you want to specify a placeholder in anOperation, you can easily do so by writingI.attributeorI['field'].Swill be replaced by the current Stream andIwill be replaced by the current Item.Ois a special case and defines a mapping for the output. If the function returns a dictionary with multiple keys and you want to extract only one of them you can write... >> {I.result: O['key_to_extract']}.RelRelis a convenience object to easily define aRelativeValue. You can writerel_val = 50%Reland it will create a newRelativeValuewith 0.5 as the value. If you then convert it to an absolute value by callingValue.to_abs(rel_val, 0, 200)it will return 100 (50% of 200).Special cases exist forRel.posandRel.neg. This will return the percentage of the positive range. If you have an array with values ranging from -100 to +50, then50%Relis -25 (exactly in the middle of -100 and +50;50%Rel.posis +25 (in the middle of the positive range: 0 to 50); and50%Rel.negis -50 (in the middle of the negative range: -100 to 0).ExampleBelow is a complete example of a pipeline.fromlyneimport*fromlyne.clipimport*target_size=(768,768)min_face_strength=10pipe=(list_dir.using(I.source_dir)|open_image|cond_size(I.image,min_size=target_size)|calc_focus|Cond(I.focus<70,'focus')|calc_lightness|Cond(I.lightness<50,'too_dark')|Cond(I.lightness>205,'too_bright')|calc_collage|Cond(I.collage>=200,'collage')|generate_attention('a photo of a face')|Cond(I.attention.max()<=0,'no_face')|scale_array(I.attention,(0,10%Rel.pos),(0,255),clip=True)>>I.attention|Cond(I.attention.mean()<min_face_strength,'small_face')|generate_attention('a photo of multiple people')|Cond(I.attention.max()>0,'multi_people')|generate_attention('a photo of a person')|Cond(I.attention.max()<=0,'no_person')|scale_array(I.attention,(0,10%Rel.pos),(0,255),clip=True)>>I.attention|add_alpha_channel(I.attention)|change_dir.using(I.path,I.target_dir)|change_ext.using(I.path,I.target_ext)|save_image)base_dir=r"D:\SD\training\Subject"item=Item(source_dir=fr"{base_dir}\orig",target_dir=fr"{base_dir}\raw",target_ext='.png',)results=pipe.process(item)#list() will iterate over the entire generatorlist(results)