Package: convey
ConveySwiss knife for mutual conversion of the web related data types, likebase64or outputs of the programswhois,dig,curl.
A convenient way to quickly gather all meaningful information or to process large files that might freeze your spreadsheet processor.Any input is accepted:if asingle valueinput is detected, allmeaningful informationis fetchedmultilinebase64/quoted_printablestring gets decodedlog/XLS/XLSX/ODS fileconverted to CSVCSV file(any delimiter, header orpandoctable format) performs one or more actionsPick, delete or sort columns(if only some columns are needed)Add a column(computes one field from another – see below)Filter(keep/discard rows with specific values, no duplicates)Split by a column(produce separate files instead of single file; these can then be sent by generic SMTP or through OTRS)Change CSV dialect(change delimiter or quoting character, remove header)Aggregate(count grouped by a column, sum...)Merge(join other file)Python3.7+ required (older releases support 3.6).Table of contentsUsageUsage 1 – Single queryUsage 2 – CSV processor programUsage 3 – Web serviceInstallation and first runPrerequisitiesLaunch as a packageOR launch from a directoryBash completionCustomisationComputing fieldsComputable fieldsWhois moduleDetectable fieldsOverview of all methodsExternal field how-toSimple custom methodLaunch an external methodRegister an external methodList of results possiblePickMethod decoratorPickInput decoratorWeb serviceSending filesArbitrary e-mail headers, "From" header, GPG signingDynamic templatesExamplesURL parsingOutput formatsComputing TLD from another columnCSV processingFile splittingCSIRT UsecaseCustom code fieldBase64 and Regular expressionsConverting unitsAggregateMergeSending images to different recipientsCreditsUsageUsage 1 – Single queryCheck what happens if an IP is provided, it returns table with WHOIS-related information and scraped HTTP content.$convey1.1.1.1# single query inputInputvaluedetected:ip1.1.1.1...au

fieldvalue
---------------------------------
prefix1.1.1.0-1.1.1.255
asnas13335
abusemailabuse@apnic.net
countryau
netnameapnic-labs
csirt-contact-
incident-contactau
status200textDNSThefreeappthatmakesyour(muchlongertext...)Usage 2 – CSV processor programParses CSV file.$conveymy-file-with-ips.csv# will trigger file parsingSourcefile:/tmp/my-file.csv
Loglines:200Sample:
...

Delimitercharacterfound:','Quotingcharacter:'"'Headerispresent:notused

Couldyouconfirmthis?[y]/n
...
...Usage 3 – Web serviceAgain, let's provide an IP to theweb service, it returns JSON with WHOIS-related information and scraped HTTP content.$convey--server# start a UWSGI sessionAccess in the browser:http://localhost:26683/?q=example.com{"ip":"93.184.216.34","prefix":"93.184.216.0-93.184.216.255","asn":"","abusemail":"abuse@verizondigitalmedia.com","country":"unknown","netname":"edgecast-netblk-03","csirt-contact":"-","incident-contact":"unknown","status":200,"text":"DNSThe free app that makes your (much longer text...)"}Installation and first runPrerequisitesYou'll be asked to installdialoglibrary at the first run if not already present in the system.If something is missing on your system, you may find help yourself with this command:sudo apt install python3-pip python3-dev python3-tk git xdg-utils dialog whois dnsutils nmap curl build-essential libssl-dev libpcre3 libpcre3-dev && pip3 install setuptools wheel uwsgi && pip3 install --upgrade ipythonbuild-essentialis needed to builduwsgiandenvelopelibpcre3 libpcre3-devneeded to suppress uWSGI warning!!! no internal routing support, rebuild with pcre support !!!libssl-devneeded to be present before buildinguwsgiif you will need to use--httpsLaunch as a package:# (optional) setup virtual environmentpython3-mvenvvenv
.venv/bin/activate(venv)$...# continue below# install from PyPipip3installconvey# without root use may want to use --user# (optional) alternatively, you may want to install current master from GitHubpip3installgit+https://github.com/CZ-NIC/convey.git# launchconvey[filenameorinputtext]# or try `python3 -m convey` if you're not having `.local/bin` in your executable pathParameter[filename or input text]may be the path of the CSV source file or any text that should be parsed. Note that if the text consist of a single value, program prints out all the computable information and exits; I.E. inputting a base64 string will decode it.OR launch from a directory# download from GitHubgitclonegit@github.com:CZ-NIC/convey.gitcdconvey
pip3install-rrequirements.txt--user# launch./convey.pyBash completionRun:apt-get install bash-completion jqCopy:extra/convey-autocompletion.bashto/etc/bash_completion.d/Restart terminalCustomisationLaunch convey with--helpflag to seefurther options.A fileconfig.iniis automatically created inuser config folder. This file may be edited for further customisation. Access it withconvey --config.Convey tries to open the file in the default GUI editor or in the terminal editor if GUI is not an option.Ifconfig.iniis present at working directory, that one is used over the one in the user config folder.Configuration is updated automatically on upgrade.Computing fieldsComputable fieldsSome field types are directly computable:abusemail– got abuse e-mail contact from whoisasn– got from whoisbase64– encode/decodecc_contact– e-mail address corresponding with the abusemail, taken from your personal contacts_cc CSV in the formatdomain,cc;cc(mails delimited by a semicolon). Path to this file has to be specified inconfig.ini » contacts_cc.country– country code from whoiscsirt_contact– e-mail address corresponding with country code, taken from your personal contacts_abroad CSV in the formatcountry,abusemail. Path to this file has to be specified inconfig.ini » contacts_abroadexternal– you specify method in a custom .py file that receives the field and generates the value for you, see belowhostname– domain from urlincident_contact– if the IP comes from local country (specified inconfig.ini » local_country) the field getsabusemail, otherwise we getcountry@@mailwheremailis eitherabusemailor abroad country e-mail. When splitting by this field, convey is subsequently able to send the split files to local abuse and foreign csirt contacts.ip– translated from urlnetname– got from whoisprefix– got from whoisWhois moduleWhen obtaining a WHOIS recordWe are internally callingwhoisprogram, detecting what servers were asked.Sometimes you encounter a funny formattedwhoisresponse. We try to mitigate such cases andre-ask another registryin well known cases.Since IP addresses in the same prefix share the same information we cache it to gainmaximal speedwhile reducingwhoisqueries.Sometimes you encounter an IP that gives no information but asserts its prefix includes a large portion of the address space. All IP addresses in that portion ends labeled as unknowns. At the end of the processing you areasked to redo unknownsone by one to complete missing information, flushing misleading superset from the cache.You may easily hitLACNIC query ratequota. In that case, we re-queue such lines to be queried after the quota is over if possible. At the end of the processing, you will get asked whether you wish to carefully and slowly reprocess the lines awaiting the quota lift.Detectable fieldsSome of the field types are possible to be auto-detected:ip– standard IPv4 / IPv6 addressescidr– CIDR notation, ex: 127.0.0.1/32port_ip– IPv4 in the form 1.2.3.4.portany_ip– IPv4 garbled in the formany text 1.2.3.4 any texthostname– or FQDN; 2nd or 3rd domain nameurl– URL starting with http/httpsasn– AS Numberbase64– text encoded with base64wrong_url– URL that has been deactivated by replacing certain chars, ex: "hxxp://example[.]com"Overview of all methodsCurrent field computing capacity can be get from--show-umlflag. Generate yours by ex:convey --show-uml | dot -Tsvg -o /tmp/convey-methods.svgDashed node: field type is auto-detectableDashed edge: field types are identicalEdge label: generating options asked at runtimeRectangle: field category borderExternal field how-toSimple custom methodIf you wish to compute anexternalfield, prepare a file whose contents can be as simple as this:defany_method(value):# do somethingreturn"modified :)"Launch an external methodWhen CSV processing, hit'Add column'and choose'new external... from a method in your. py file'Or in the terminal append--field externalto yourconveycommand. A dialog for a path of the Python file and desired method will appear.$convey[string_or_filepath]--fieldexternalYou may as well directly specify the path and the callable. Since the--fieldhas following syntax:FIELD[[CUSTOM]],[COLUMN],[SOURCE_TYPE],[CUSTOM],[CUSTOM]You may omit bothCOLUMNandSOURCE_TYPEwriting it this way:FIELD,COLUMN,SOURCE_TYPE,CUSTOM,CUSTOMexternal,/tmp/myfile.py,any_method$convey[string_or_filepath]--fieldexternal,/tmp/myfile.py,any_method
Inputvalueseemstobeplaintext.
fieldvalue
-------------------------------
externalmodified:)Register an external methodYou may as well hard code custom fields in theconfig.iniby providing paths to the entry point Python files delimited by a comma:external_fields = /tmp/myfile.py, /tmp/anotherfile.py. All the public methods in the defined files will become custom fields![EXTERNAL]
external_fields = /tmp/myfile.pyIf this is not needed, you may register one by one by adding new items to theEXTERNALsection. Delimit the method name by a colon.[EXTERNAL]
any_method = /tmp/myfile.py:any_methodList of results possibleIf you need a single call to generate multiple rows, return list, the row accepting a list will be duplicated.defany_method(value):# do somethingreturn["foo","bar"]When convey receives multiple lists, it generates a row for each combination. Ex: If a method returns 2 items and another 3 items, you will receive 6 similar rows.PickMethod decoratorShould there be multiple ways of using your generator, place them as methods of a class decorated withPickMethodand let the user decide at the runtime.PickMethodhas optionaldefault:strparameter that specifies the default method.fromconveyimportPickMethod@PickMethod("all")classany_method(PickMethod):defall(x):''' All of them.  '''returnxdeffiltered(cls,x):''' Filter some of them '''ifxincountry_code_set:returnx$conveyfile.csv--fieldany_method# user will be asked whether to use `all` or `filtered`$conveyfile.csv--fieldany_method[filtered]# `filtered` sub-method will be used$conveyfile.csv--fieldany_method--yes# the default `all` sub-method will be usedPickInput decoratorIf you need a direct user entry before each processing, importPickInputand make your method accepttwoparameters. The latter will be set by the user and may have a default value.fromconveyimportPickInputimportdateutil@PickInputdeftime_format(val,format="%H:%M"):''' This text will be displayed to the user.If running in headless mode, the default format will be "%H:%M" (hours:minutes).   '''returndateutil.parser.parse(val).strftime(format)$conveyfile.csv--fieldtime_format# user will be asked for a format parameter$conveyfile.csv--fieldtime_format[%M]# `format` will have the value `M%`$conveyfile.csv--fieldtime_time--yes# the default `format` `%H:%M` will be usedWeb serviceWhen launched as a web service, several parameters are available:q– search querytype– same as--typeCLI flagfield– same as--fieldCLI flag.Note that unsafe field typescodeandexternalare disabled in web service by default. You may re-allow them inwebservice_allow_unsafe_fieldsconfig optionfull syntax of CLI flag is supportedFIELD[[CUSTOM]],[COLUMN],[SOURCE_TYPE],[CUSTOM],[CUSTOM]Ex:reg_s,l,Lperforms regular substitution of'l'by'L'clear=web– clears web scrapping module cacheQuick deployment may be realized by a single command:$convey--serverNote that convey must be installed viapipNote that LACNIC may freeze for 300 s, hence the timeout recommendation.Note that you may find your convey installation path by launchingpip3 show conveyIf you received orgeneratedkey and certificate, you may turn on HTTPS inuwsgi.iniaccesible by:convey --config uwsgiInternally, flag--serverlauncheswsgi.pywith a UWSGI session.$uwsgi--http:26683--http-timeout310--wsgi-file/home/$USER/.local/lib/python3.../site-packages/convey/wsgi.pyAccess:curl http://localhost:26683/?q=example.com{"ip":"93.184.216.34","prefix":"93.184.216.0-93.184.216.255","asn":"","abusemail":"abuse@verizondigitalmedia.com","country":"unknown","netname":"edgecast-netblk-03","csirt-contact":"-","incident-contact":"unknown","status":200,"text":"DNSThe free app that makes your (much longer text...)"}Access:curl http://localhost:26683/?q=example.com&field=ip{"ip":"93.184.216.34"}Access:curl http://localhost:26683?q=hello&type=country&field=reg_s,l,L{"reg_s":"heLLo"}Sending filesWhen you split the CSV file into chunks by an e-mail, generated files may be sent to these addresses. Look at the example of an unlocked "send" menu below. You see the list of the recipients, followed by a conditional list of recipients that have been already sent to. Next, an exact e-mail message is printed out, including headers.In the menu, you may either:Sendthe e-mailsLimitthe messages that are being send at once; if you are not 100 % sure you want to send the the whole message bucket at once.Editthe template. The message file will open either in the default GUI or terminal editor. The first line of the template should beSubject: ..., followed by a free line. Note that you may include any e-mail headers, such asReply-To: ...,Cc: ..., etc. The e-mail will reflect all of them. You may write the message either in plain text or in the HTML.Choosewhich recipients in a checkbox list will receive the message.Testsending a message to your own address. You'll be prompted which of the messages should be delivered to you. The e-mail contents possibly modified by a dynamic template is shown just before sending.Print all e-mailsto a file to have the more granulated control over what is going to be sent.Toggle file attachingon and off. A portion of the source file related to this e-mail address might be attached. There are use cases when you do not want the files to be sent with as the body text suits fine.Toggle paths from the path column attachingon and off. Files mentioned in a path column and related to this e-amil address might be attached.***E-mailtemplate***
Recipientlist(1/3):alice@example.com
Alreadysent(2/3):bob@example.com,cilia@example.com
Attachment:splitCSVfileattached

Content-Type:text/plain;charset="utf-8"Content-Transfer-Encoding:7bit
MIME-Version:1.0
Subject:Mysubject
From:me@example.com
Date:Fri,17Jan202001:36:28+0100

Hello,

thisismytestingmessage.

Keenregards

**************************************************1)Sendalle-mailsvialocalhost(1)←←←←←
l)Limitsendingamountto...
e)Edittemplate...
r)Chooserecipients...
t)Sendteste-mail...
p)Printe-mailstoafile...
a)Attachfiles(toggle):True
i)Attachpathsfrompathcolumn(toggle):False
x)Goback...
?Arbitrary e-mail headers, "From" header, GPG signingIn the template, you may specify any e-mail header, such asReply-To,CcorFrom. IfFromis not found, we takeSMTP/email_from_nameconfig value. Ifgnupghome is found on the default user path, we check if there is a secret key matching theFromheader and if found, e-mail will be GPG-signed. If it is going to be signed, you would see something likeContent-Type: multipart/signed; protocol="application/pgp-signature";header in the e-mail template preview.Dynamic templatesMessage is processed withJinja2templating system by default.Few instruments are included to treat the attachment contents:attachment()– Prints the attachment contents and prevent it to be attached.You will find our findings below.

{{ attachment() }}row()– Generate attachment contents fields row by row. Header skipped.amount(count=2)– Check if the attachment has at leastcountnumber of lines. Header is not counted. Useful when deciding whether the are single row in the result or multiple.joined(column: int, delimiter=", ")– Return a column joined by delimiterfirst_row– Access first line fieldsExample:{% if amount() %}
    Here is the complete list of the elements.
    {{ joined(1,"\n") }}
{% else %}
    Here is the element you had problems with: {{ first_row[1] }}
{% endif %}ExamplesIn the examples, we will use these parameters to add a field and to shorten the result.# -f, --field adding field syntax: FIELD[[CUSTOM]],[COLUMN],[SOURCE_TYPE],[CUSTOM],[CUSTOM]# -H, --headless: just quietly print out single value, no dialogURL parsingOutput formatsPut any IP or URL as the argument.$conveyexample.com
Inputvaluedetected:hostname

Whois93.184.216.34...abuse@verizondigitalmedia.com
Scrappinghttp://example.com...
fieldvalue
----------------------------------------------------------------------------------------------
cidr93.184.216.0/24
ip93.184.216.34
tldcom
urlhttp://example.com
abusemailabuse@verizondigitalmedia.com
csirt_contact-
incident_contactabuse@verizondigitalmedia.com
netnameedgecast-netblk-03
prefix93.184.216.0-93.184.216.255
a93.184.216.34
aaaa2606:2800:220:1:248:1893:25c8:1946
mx0.
ns['a.iana-servers.net.','b.iana-servers.net.']spfv=spf1-all
http_status200textExampleDomainThisdomainisforuseinillustrativeexamplesindocuments.Youmayusethisdomaininliteraturewithoutpriorcoordinationoraskingforpermission.Moreinformation...Should you need just the country the domain/IP is hosted in, use--field, -fargument$conveywikipedia.com-fcountry
Inputvaluedetected:hostname

Whois208.80.154.232...us
fieldvalue
--------------
countryusUse--headless, -Hor--quiet, -qflag to shorten the output (and cut down all dialogues).$conveywikipedia.com-fcountry-H
usFlag--jsonmodifies the output.$conveywikipedia.com-fcountry-H--json{"country":"us"}Computing TLD from another columnTo compute a TLD from the abusemail that is being used for the IP domain is hosted in, add a fieldabusemailand then another fieldtld. Specifically say that the latter should source from the second column (which isabusemail) – either type '2' or 'abusemail'.$conveyexample.com-fabusemail-ftld,2
$conveyexample.com-fabusemail-ftld,abusemail
Inputvaluedetected:hostname

Whois93.184.216.34...abuse@verizondigitalmedia.com
fieldvalue
--------------------------------------
abusemailabuse@verizondigitalmedia.com
tldcomTo preventabusemailfrom being output, use--field-excluded, -feinstead of--field, -f:$conveyexample.com-feabusemail-ftld,2-H
Inputvaluedetected:hostname

Whois93.184.216.34...abuse@verizondigitalmedia.com
fieldvalue
--------------
tldcomWe did not say earlier, user is asked each time whether they wish to get anytld,gTLD(ex:com) orccTLD(ex:cz). You may specify it from CLI by one of those equivalent commands.$conveytest.csv--fresh--fieldtld[gTLD]$conveytest.csv--fresh--fieldtld,,,gTLD# flag --yes or --headless will choose the default option which is *all*$conveytest.csv--fresh--fieldtld--yesCSV processingShould you have a list of the object that you want to enrich of a CIDR they are hosted at, load the filetest.csvthey are located in.# file text.csv
domain list
wikipedia.com
example.comAnd see the menu just by adding--field cidrargument.$conveytest.csv-fcidr
Sourcefile:/tmp/ram/test.csv
Identifiedcolumns:
Loglines:3Sample:
domainlist
wikipedia.com
example.com

Delimitercharacterfound:','Quotingcharacter:'"'Headerispresent:yes

Couldyouconfirmthis?[y]/n:(HITENTER)Sourcefile:/tmp/ram/test.csv,delimiter:',',quoting:'"',header:used
Identifiedcolumns:domainlist(hostname)Computedcolumns:cidr(fromdomainlist)Loglines:3Sample:
domainlist
wikipedia.com
example.com

Whois208.80.154.232...us
Whois93.184.216.34...abuse@verizondigitalmedia.com
Preview:
domainlistcidrfrom:(hostname)domainlist
------------------------------
wikipedia.com208.80.152.0/22
example.com93.184.216.0/24

Mainmenu-howthefileshouldbeprocessed?1)Pickordeletecolumns2)Addacolumn3)Filter4)Splitbyacolumn5)ChangeCSVdialect6)Aggregate7)Merge
p)process←←←←←
~)send(splitfirst)~)showalldetails(processfirst)r)redo...
c)config...
x)exit?File splittingWe will create an ASN field and split the file.csv by this field, without adding it into the output.# file.csv
wikipedia.com,443,2016-02-09T01:12:26-05:00,16019,US
seznam.cz,25,2016-02-27T22:20:21-05:00,16019,CZ
google.com,25,2016-02-28T02:27:21-05:00,16019,US$conveyfile.csv--field-excludedasn--splitasn(...)**Processingcompleted:3resultfilesin/tmp/ram/file.csv_convey1573236314(...)# file as14907
wikipedia.com,443,2016-02-09T01:12:26-05:00,16019,US# file as43037
seznam.cz,25,2016-02-27T22:20:21-05:00,16019,CZ# file as15169
google.com,25,2016-02-28T02:27:21-05:00,16019,USCSIRT UsecaseA CSIRT may use the tool to automate incident handling tasks. The input is any CSV we receive from partners; there is at least one column with IP addresses or URLs. We fetch whois information and produce a set of CSV grouped by country AND/OR abusemail related to IPs. These CSVs are then sent by through OTRS from within the tool.
A most of the work is done by this command.convey--field-excludedincident_contact,source_ip--splitincident_contact--yes[FILENAME]Custom code fieldAdding a column from custom Python code:$conveyexample.com-fcode,"x=x[1:5]"xampBase64 and Regular expressionsCode there and back:$conveyhello-fbase64-H# --headless conversion to base64aGVsbG8=$conveyaGVsbG8=-H# automatically identifies input as base64 and produces plaintexthelloUse aregcolumn for regular expressions.# start adding a new reg column wizzard that will take decoded "hello" as input$conveyaGVsbG8=-freg
$conveyaGVsbG8=-freg_s,"ll","LL"-H# substitute 'll' with 'LL'heLLoSpecify source# start adding a new reg column wizzard that will take plaintext "aGVsbG8=" as input$conveyaGVsbG8=-freg,plaintext# specifying plaintext as a source type will prevent implicit convertion from base64$conveyaGVsbG8=-freg_s,plaintext,"[A-Z]","!"-H# substitute uppercase letters with '!'a!!sb!8=Converting unitsWe are connected to thepintunit converter!$convey"3 kg"Inputvaluedetected:unit

fieldvalue
-----------------------------------------------------------------------------------------------------
plaintext['1.806642538265029e+27 atomic_mass_unit','105.82188584874123 ounce','96.45223970588393 apothecary_ounce','0.0703602964419822 bag','0.05905239165666364 long_hunderweight','0.06613867865546327 US_hundredweight','0.002952619582833182 UK_ton','0.002952619582833182 long_ton','1929.0447941176785 pennyweight','46297.07505882429 grain','1.7935913792661326e+27 proton_mass','771.6179176470714 apothecary_dram','3000.0 gram',...]$convey"3 kg"-funit# launches wizzard that let's you decide what unit to convert to$convey"3 kg"-funit[g]-H3000.0gram

$convey"kg"-funit--csv-processing--headless
kg|6.022141794216764e+26atomic_mass_unit
kg|0.001metric_ton
kg|0.0009842065276110606UK_ton
kg|771.6179176470714scruple
kg|257.2059725490238apothecary_dram
kg|1000.0gram# You may try to specify the units with no space and quotation.# In the following example, convey expand all time-units it is able to compute# – time units will be printed out and each is base64 encoded.$convey3hours
Inputvaluedetected:timestamp,unit

fieldvalue
-------------------------------------------------------------------------------------(...)base64['MC4wMDQxMDY4NjM4OTc0NTAwNzcgbW9udGg=','MTA4MDAuMCBzZWNvbmQ=','MC4wMTc4NTcxNDI4NTcxNDI4NSB3ZWVr']plaintext['0.004106863897450077 month','10800.0 second','0.01785714285714285 week',(...)]time03:00:00(...)# What if$convey3hours-furlencode
Inputvaluedetected:timestamp,unit

Inputunit_expandvariableunit:*youtypeheresecorsecondstoseethewizzard*
Preview|original|result||------------|----------------||3hours|10800.0second|fieldvalue
-------------------------
urlencode10800.0%20second# What if we wanted to urlencode text "3hours" without converting it to unit first?# Just specify the SOURCE_TYPE to be plaintext:$convey"3hours"-furlencode,plaintext
Inputvaluedetected:timestamp,unit

fieldvalue
----------------
urlencode3hoursCountry namesWhen specifying country code, you get country name.$convey--typecountrysc
SeychellesLots of countries can be recognized.bash
$ convey --type country_name Futuna
wfYou may get the country code from various telephone number formats.$convey+2481234567
Seychelles

$convey"1-541-754-3010"['ca','us']$conveyAggregateSyntax is[COLUMN, FUNCTION], ..., [group-by-COLUMN].
Possible functions are:avgsumcountminmaxlistsetLet's have a file.# file.csv
category,price,consumption
bulb,100,30
bulb,150,10
kettle,250,70
kettle,352,80
bulb,120,15Sum thepricecolumn.$conveyfile.csv--aggregateprice,sumsum(price)------------972Group thepricesum bycategory.$conveyfile.csv--aggregateprice,sum,category
categorysum(price)---------------------
total972bulb370kettle602Group thepricesum and theconsumptionaverage value bycategory.$conveyfile.csv--aggregateprice,sum,consumption,avg,category
categorysum(price)avg(consumption)----------------------------------------
total97241bulb37018.33
kettle60275Group thepricesum bycategoryand list its values.$conveyfile.csv--aggregateprice,sum,price,list,category
categorysum(price)list(price)-------------------------------------------
total972(all)bulb370['100','150','120']kettle602['250','352']You can even split while aggregating. Each file will count its own results.$conveyfile.csv--aggprice,sum--splitcategory

Splitlocation:bulbsum(price)------------370Splitlocation:kettlesum(price)------------602MergeYou can merge two file based on a common column.Imagine a XLS fileperson.xls, containing following rows:john@example.com,foo,male
mary@example.com,foo,female
hyacint@example.com,bar,maleAnd a filesheet.csvfoo,red,second.example.com
foo,green,first.example.com
bar,blue,wikipedia.org
bar,yellow,example.com
foo,orange,wikipedia.comGet them merged with a single command. We specify the merged file will beperson.xls, while the common column (foo/bar) is the second in the remote and the first in the localsheet.csvfile.$convey--output--headless--filesheet.csv--mergeperson.xls,2,1
foo,red,second.example.com,mary@example.com,female
foo,red,second.example.com,john@example.com,male
foo,green,first.example.com,mary@example.com,female
foo,green,first.example.com,john@example.com,male
bar,blue,wikipedia.org,hyacint@example.com,male
bar,yellow,example.com,hyacint@example.com,male
foo,orange,wikipedia.com,mary@example.com,female
foo,orange,wikipedia.com,john@example.com,maleWhen a pivot key (local column value)is missing from the remote file, the fields on the line stays blank.is found twice, the row gets duplicated.Performance note: Working with really huge files? Whereas the local file that you merge to can be of an arbitrary size, the remote file being merged should not be excessively big, it should fit to the RAM.Sending images to different recipientsImagine you have a directory full of PNG files, containg info for respective domain administrators.example.com.png
example.com-2.png
wikipedia.org.png
csirt.cz.pngWe want to send two images to the admin ofexample.comand single one forwikipedia.organdcsirt.cz. First, create a CSV having first column the domain name and second the respective image. To do so, we use thepzlibrary.pip3 install pz  # install `pz` if needed
ls *.png | pz 're.sub("-\d+$", "", Path(s).stem),s' > domains_images.csvCSV domains_images.csv now looks like this:example.com	example.com-2.png
example.com	example.com.png
wikipedia.com	wikipedia.com.png
csirt.cz	csirt.cz.pngThe following command will tellconveyto add an e-mail column derived from the hostname and when sending files, attach all our images from the path column.convey domains_images.csv  -t hostname,path -f abusemail --split abusemail --attach-files False  --attach-paths-from-path-column TrueWizzard will lead you to sending the test mail to you own address. This is the sending a testing e-mail dialog for thewikipedia.orgdomain.Attachment wikipedia.com.png (image/png): PNG...
MIME-Version: 1.0
Subject: PNG that might interest you
To: abuse@wikimedia.org
Date: Fri, 06 Jan 2023 20:04:31 +0100

Body text message
Testing e-mail address to be sent to – type in or hit Enter to use your-testing-mail@example.com (Ctrl+C to go back):CreditsBrought byCSIRT.cz.